Page  00000122 SONOFUSION: DEVELOPMENT OF A MULTIMEDIA COMPOSITION FOR OVERTONE VIOLIN John Thompson Dan Overholt University of California, Santa Barbara Media Arts and Technology Program Santa Barbara, California, USA ABSTRACT This document describes Sonofusion, the debut multimedia content created for the Overtone Violin, and our collaborative development of an audiovisual performance system for the instrument and the work. The Overtone Violin was originally developed to support both improvisatory and deterministic compositions, and Sonofusion is the first largely deterministic work to be composed specifically for the instrument [10]. Design considerations and implementation details of both the performance system and the composition are described in detail. 1. INTRODUCTION There exist a growing number of augmented instruments for musical performance. Many of these instruments are explored freely within the context of improvisation. Unfortunately, there is a lack of formal investigation of these instruments through the composition of more deterministic content. Perhaps because of this, many augmented instruments quickly become novelty items that do not stand the test of time. By working collaboratively as performer/instrument builder and composer, we attempt to remedy this situation. Aside from a variety of improvisatory performances, few compositions have been written for the Overtone Violin, and these are only partially notated, relying heavily upon improvisation. Through our collaboration the audiovisual composition Sonofusion arose. Sonofusion is a largely deterministic work written specifically for the Overtone Violin. The work creates a notational system for the instrument and establishes a software system for the composition of audiovisual works using the instrument. In this paper we describe the performance system developed for Sonofusion. 2. BACKGROUND Many interactive instruments have been created to gather gestural control from the performer. These instruments range from extensions of familiar preexisting instruments to novel instruments that lend gestural capabilities to objects not normally associated with musical production. We focus here on the former, also called augmented or hybrid instruments. These expand on an existing traditional instrument, becoming evolutionary enhancements to an interface rather than revolutionary "start from scratch" approaches. We see this as a beneficial way to "stand on the shoulders" of those before us in both performance and compositional realms. Golan Levin's "Audiovisual Environment Suites" [7] make excellent use of simple computer mouse controls. Although practical in a utilitarian sense, the computer keyboard and mouse do not provide very expressive affordances for the performer, or give an overtly dramatic connection between performance gesture and the perceived result. They also do not build on any existing musical instruments as a way to upholding the strengths of an evolved musical interface and expanding on a musician's instrumental skills. Dan Trueman and Curtis Bahn's electroacoustic performance group Interface utilizes several augmented instruments and alternative controllers such as the SBass [1] and the Bowed-Sensor Speaker Array (BoSSA) [9]. While the SBass functions as a true augmented instrument, BoSSA functions as an alternate controller since it does not afford the functions of a traditional violin (e.g. it lacks strings). Other augmented instruments include the Hyperbow [12] at the MIT Media Lab, and the Augmented Cello project [4] at U.C. Berkeley's Center for New Music and Audio Technologies. Instrument builders who perform with augmented instruments often develop their own performance systems as well. While this is a productive method for the motivated individual, it does not facilitate the composition of works for the instrument by the musical community. This individualistic method of working stands in the way of allowing augmented instruments to find a wider base of users, and ultimately, a way of finding a lasting place in the musical world. The authors have found many benefits of collaborative work on both practical and conceptual levels through this work. 3. THE OVERTONE VIOLIN The Overtone Violin [8] is an entirely custom built, radically augmented musical instrument that preserves the traditions of violin technique while adding an entirely new set of possibilities for the performer. It can be seen as two components tightly integrated into one, in that it is both a traditional electric violin, and a gestural computer music controller (see Figure 1). As an augmented instrument, the Overtone Violin offers an extensive set of controls. These controls include both continuous and discrete sensors. The discrete sensors consist of sixteen buttons on the violin scroll and an additional two buttons above the fingerboard on the base of the scroll (parallel to the 122

Page  00000123 strings). These discrete controls send information in the form of on-off toggle commands. In contrast to the discrete controls, the continuous controls offer dynamic gestural control. Figure 1. The right hand glove worn by the performer contains an x-y accelerometer and a sonar transducer that broadcasts pings to a set of receivers near the bridge of the violin. Using these sensors, the Overtone Violin expresses a number of functional gestures already familiar to traditional violin players. These include twodimensional bow rotation, bow speed, and bow position. These controls can be used in the course of most violin playing techniques or as stand-alone extensions to traditional techniques. The additional continuous controls are less familiar to the traditional violin player and consist of a joystick, two sliders, and a thumb slider on the back of the scroll. These controls provide extended techniques beyond normal violin methods, and as such the violinist is unable to simultaneously use his left hand on the fingerboard. This also applies to the use of the three data knobs located on the side of the violin. Two additional sensors are available, a second sonar transducer and a video camera, both located on the violin's scroll. The sonar transducer ultrasonically measures the distance between the violin and any object, such as a music stand or another person. The video camera wirelessly transmits both image and sound, which can be used by computer vision and audio analysis techniques. The rationale behind the development of the instrument was to keep the expressive elements of the expert violinist, while incorporating the added benefits of extended gestural controllers via embedded sensors. Any instrument can be augmented to various degrees through the addition of electronic sensors; hybrid instruments offer musicians the familiarity and expressivity of their chosen instrument along with the extended control of digital algorithms afforded by the sensors. There are two ways, however, in which the Overtone Violin differs from most hybrid instruments. First, in addition to acquiring traditional violin techniques, the extra sensors are used to capture an extended set of gestures that the performer must learn, thereby augmenting playing methods. Second, it is designed and built from scratch to be an entirely new, specialized instrument that continues the evolution of the violin, rather than retrofitting an older instrument. The philosophy of this approach is to use gestural sensors to add new functionality to the instrument in addition to capturing traditional playing techniques with as much nuance as possible. With the Overtone Violin, real-time signal processing of audio synthesis or effects is under direct expressive control of the performer. This leads to an open-ended set of possibilities for the musician, and lends itself well to both improvisatory settings and more deterministic compositions that can be practiced accurately and reproduced and interpreted with as much expressivity as can be achieved by the effort of each performance. 4. SONOFUSION: FOR OVERTONE VIOLIN The Overtone Violin does not have fixed audio or visual mappings, thus software systems must be designed for compositions or improvisations. The goals of the software system used for Sonofusion were: A. Create a largely deterministic interactive piece for the Overtone Violin. B. Use cross-modal techniques to create a tight coupling between audio, visuals, and gesture. C. Let the performer control the interactivity through the augmented instrument techniques. 4.1. The software engine The underlying software engine created for the Overtone Violin's performance system is constructed to accommodate audiovisual works that feature multiple virtual spaces linking audio and visual media. The system accommodates this linkage from the level of audio signals driving the visuals, down to the level of shared synthesis parameters between audio and visual synthesis and transformation algorithms. The software engine uses Max/MSP/Jitter [2] working in tandem with SuperCollider [3] through Open Sound Control in a potentially distributed setup. Max/MSP/Jitter is used for graphics display as well as for interfacing the instrument. Data from the instrument is sent to SuperCollider, which performs audio synthesis, transformation (effects processing), and contains the underlying compositional routines. This system is extensible and its logic can be uplifted for the creation of additional works. The Overtone Violin transmits extremely lowlatency control signals and audio signals wirelessly allowing technical transparency in the performance. It was important to us to maintain that transparency and allow the software system to perform without supervision. A state machine was constructed to help eliminate performance errors that result from errant toggle commands, either from extraneous wireless sensor data transmission, or errors from the performer. If deemed valid by the state machine, the toggle command initiates a virtual space in Max/MSP/Jitter and this triggers routines, instantiates synthesis algorithms, and reroutes the mapping scheme for gestural signals according to the compositional scheme in SuperCollider. Data flow in the opposite direction (from SuperCollider to Max/MSP/Jitter) enables the parametric data of audio synthesis algorithms to drive the parameters of visual algorithms. This is necessary to 123

Page  00000124 create cross-modal ties between the media at a low level. Parametric data from audio synthesis algorithms is projected into two-dimensional image and threedimensional virtual spaces creating deep structural connections between the media [6]. To create surface level connections, audio signals from the violin and synthesis events are used to drive the visual system. 4.2. Creating deterministic content The rigorous constraints of creating a repeatable work for the Overtone Violin with defined gestural expectations added a refreshing degree of clarity to the process of writing for an augmented instrument. While the performances can vary there is always a ground truth. In the creation of the work, this ground truth was crucial to improving the mappings for the physical instrument and software system. 4.3. Mappings in flux Static mappings that connect gesture to transformation parameters can quickly become tiring. A flexible network of gesture-to-parameter mappings, changed during the course of the piece, thwarts what could be a monotonous tie between a single transformation parameter and a single gesture. To create a flexible mapping paradigm, gestural controls are streamed onto control busses in SuperCollider. These busses can be accessed at any time and scaled and offset in various ways. This affords a mapping situation that isn't fixed but can be dynamically adjusted for various sections and sub-sections of the work. The network of gesture-to-parameter mappings can be extended to interaction with a collection of quasiautonomous routines. Rather than providing direct control of sonic transformations, interaction with this collection of routines provides a layer of indirection between the performer and the computer-generated result. While these routines generate synthesis and transformation parameters algorithmically, parameters of the routines are made available to the performer to control interactively. Performer interaction with these parameters allows for one-to-many mappings between the gesture and the parameters of the synthesis and transformation processes. In addition to preventing trivial and tiring mappings, the indirection provided by the interaction with the routines prevents the performer from being overwhelmed by an overly heavy cognitive% load [5]. Mapping is seen as a space where both artistic and engineering approaches can be taken. The authors feel that if the end result of a project is intended to be artistic, mapping should be considered to be part of the compositional process rather than entirely an engineering challenge. Below is a brief synopsis of the general mapping characteristics in each section of Sonofusion. Interested readers can find video and a score of the work at [11]. Section 1: Audio amplitude from the Overtone Violin is mapped to lens angle. The louder the sound the smaller the lens angle (the closer the zoom) IA. Buttons located at the top of the fingerboard are used to trigger sample playback routines. Audio amplitudes from these samples are mapped to movie playback position. Section 2: The performer triggers new sections with a button. In section 2, parameters of audio trainlets are mapped to vertices of an unfolding geometric shape. These unfold generatively independently of the Overtone Violin. Bow rotation controls the color parameters and bow position controls the radius parameter of the geometry-defining algorithm. The joystick of the Overtone Violin controls parameters of granular routines performed on samples. 2A. Bow position is remapped (via a button) to the delay time of a comb filter. As the bow moves further away from the violin, the delay time grows shorter. These delay times are small enough to produce pitches, thus producing a glissando as the bow moves outward. Additionally the radius of the geometric shape is tied to the bow's position. A pizzicato gesture creates a sonic glissando with the delay line and an outward spiraling of the geometric shape; gesture is connected across audio and visual media. Section 3: Individual agents wander in a bounded space. Each agent has possesses pitch and rhythmic 124

Page  00000125 characteristics. Agents measure the distance to a central agent and when within a certain threshold, they sound their pitch and rhythmic contents and visualize their activity. The performer controls the virtual camera rotation by triggering preset paths with button presses. At most times in this section audio is mapped to an envelope following filter. 3A. In response to the glissando produced by outward bow movements in section 2, a pitch shift (granular) is mapped to bow position in parts of this section, again triggered by the press of a button. The pitch shift is mapped to the range of an octave from the minimum bow position to the maximum. The performer bows a note and a double stop extends an octave above at the rate of the downbow. 3B. Throughout this section sampled sounds (spectrally transformed violin recordings) are triggered via button presses. Section 4: Bow position is quantized into five regions. Within each region, bow position and rotation control different parameters (or different scaling and offsets of parameters) of a granulation routine. Visually, an image is textured onto a shape. Audio amplitude (if rate of change is greater than a given threshold) is mapped to control the parameters of a visual algorithm that disperses pixels of the texture based on luminance (jit.plume). Amplitude is also controls the rotation of the shape upon which the image is textured. Bow rotation controls the color parameters of the image. 4A. Simple comb delays are initiated through button presses throughout the section. Section 5: In the final section, audio amplitude controls parameters of a generated shape as well as the scale of the shape and the alpha channel of its color. 5A. Through the use of a knob on the Overtone Violin, the grain length parameter of a granular routine that underlies the section is increased. 5. CONCLUSIONS The Overtone Violin proves to be an instrument with great flexibility. The extensive set of controls provides an abundance of possibilities for performative control of audiovisual processes. Through this collaborative work, we created a performance system for interactive audiovisual performances, a notational system, a new lexicon of performance techniques for the Overtone Violin, and the beginning of a deterministic repertoire for the instrument. Our hope is that by establishing this performance system, we facilitate the creation of additional works for the Overtone Violin by various composers. A collaborative approach to creating performance systems for augmented instruments suits the needs of not only the instrument builder/performer but also the composer who desires to use the instrument to create works. This affords a better opportunity for augmented instruments to make their way toward canonization in the larger musical community. 6. REFERENCES [1] C. Bahn, SBass - sensor bass interface, accessed May 2007. [2] Cycling74. accessed May 2007. [3] The SuperCollider Home Page, accessed May 2007. [4] Freed, A., Wessel, D., Zbyszynski, M., and Uitti, F., "Augmenting the Cello", Proceedings of the 2006 NIME conference, pp. 409-413. [5] I. Choi and R. Bargar, and C. Goudeseune, "A Manifold Interface for a High Dimensional Control Space", Proceedings of the 1995 ICMC, pp. 385-392. [6] R. Dannenberg, "Interactive Visual Music: A Personal Perspective", Computer Music Journal, Volume 29 (Winter 2005), No. 4, pp. 25-35. [7] G. Levin, "Painterly Intefaces for Audiovisual Performance", M.S. Thesis, MIT Media Laboratory, 2000. [8] D. Overholt, "The Overtone Violin, a New Computer Music Instrument" in Proceedings of the 2006 ICMC, 2006. [9] D. Trueman and P. Cook, "BoSSA: The Deconstructed Violin Reconstructed", Journal of New Music Research, Issue 29 (2000), No 2, pp. 121-130. [10] J. Thompson, "Aesthetic and Technical Considerations Integral to the Composition of Sonofusion for Overtone Violin", Ph.D. Dissertation, UC Santa Barbara, 2006. [11] Sonofusion, htti:wwwtinmebentucor/sono fsni! accessed July 2007. [12] Young, D. and Nunn, P. and Vassiliev, A. "Composing for Hyperbow: A Collaboration Between MIT and the Royal Academy of Music." Conference on New Interfaces for Musical Expression (NIME). Paris, 2006. 125