Page  00000488 Fractured Sounds, Fractured Meanings: A Glove-Controlled Spectral Instrument David Litke School of Music University of British Columbia 6361 Memorial Dr. Vancouver, B.C. CANADA V6T 1Z2 litkedavid@hotmail.com ABSTRACT The compositional practice of applying data collected from FFT analyses to pre-compositional organization has been well established by composers such as Gerard Grisey, Tristan Murial, and Kaija Saariaho. The instrument presented here builds upon the fundamental ideas of these spectral composers, applying the concepts of timbral disintegration and re-synthesis in a real-time environment. By allowing a performer to sample a source sound, de-construct its overtone spectrum, and manipulate its individual components in a live performance environment, the Spectral Instrument facilitates the perception and comprehension of spectral concepts for an audience. In order to afford performers expressive, gestural control over spectral materials, the instrument uses a three-dimensional glove controller as its primary interface. In using the instrument to deconstruct and manipulate the overtones of the human voice, interesting parallels emerge between the semantic structures of language and timbral identities. 1. OVERVIEW The Spectral Instrument is a Max/MSP [3] - based electronic instrument that allows the user to differentiate and manipulate the individual spectral components of a sound sample in a live performance, using a three-dimensional glove controller. The source sound used by the instrument can either be sampled from an instrument or voice during performance, or read from a file. By allowing independent control over the source sound's partials, the instrument enables the performer to heighten the listener's perception of the sound's spectral characteristics, making audible connections between timbral characteristics and harmonic structures. After an initial recording phase, during which the incoming signal is parsed and stored, the performer can manipulate the sound materials to create a wide range of textures and timbres. Furthermore, the data collected during partial tracking can be transferred to other modules for use in creating new sounds, generating musical materials for composition or improvisation, or controlling other processes. While the source sounds that the Spectral Instrument best accommodates tend to have relatively stable and strong partials, the frequency of these partials can vary over time without posing problems; inharmonic sounds, such as those made by bells and gongs, can be used as readily as harmonic sounds such as the voice or the violin. Of particular artistic interest are the symbolic parallels drawn in using the instrument with vocal sounds; capturing and repeating a fragment of a vocal line begins a process of deconstruction of both the text and the musical phrase, and this process is continued when the acoustic constitution of the sound is itself deconstructed. 2. HISTORICAL BACKGROUND In spectrally-informed compositional practice, composers have tended to apply data from FFT analyses primarily to the pre-compositional aspects of their pieces. Seminal works such as Gerard Grisey's Partiels [1] have set a precedent for this approach, in which structures found within a sound sample are abstracted, providing raw material for application in compositional structures. This paradigm continues to be fruitful for composers, providing a conceptual link between acoustic phenomena and the organization of sounds into musical compositions. However, an emphasis on pre-compositional application of spectral concepts has tended to overshadow the possibilities of using the same ideas in a quasi-improvisatory performance setting. By electronically deconstructing or re-constructing a source sound in a live performance, the aural connections between the overtone structure of a particular timbre and the individual pitches and intensities of each partial are elucidated for the audience, facilitating comprehension of the connections between the music's micro and macro structures. Although real-time manipulation of the spectral characteristics of a sound is common, the range of techniques is generally limited to various types of filtering or cross-synthesis. In these methods, the modified spectrum remains fused as a single sonic 488

Page  00000489 object, and is perceived as an altered version of the original source sound(s). The spectral processing described here is aimed at isolating the individual components of a source sound, in order to explore the boundaries between the perception of a unified timbre and that of an amalgam of discrete pitches. 3. GESTURAL CONTROL Real-time electronic controllers face the challenge of how to achieve subtle, varied, and expressive control over sound production. To this end, a threedimensional glove interface (the P5 gaming glove, shown in Fig. 1) with finger articulation sensors was chosen for control of the Spectral Instrument. This interface affords the performer simultaneous control over four dynamic parameters, which are intuitively synthesized by gestures of the hand in space. While other control methods could have been used to control these dynamic variables (such as a set of four sliders, for example), the glove controller allows the performer to easily generate complex interactions between the four parameters while retaining a simple conceptual image of the gesture needed to create those interactions. The performer is able to mentally construct a three-dimensional "sonic space" that can be navigated; since each region of the space corresponds to a different quality of sound, the performer need only reach for the desired sonic area rather than consider how each of the individual parameters needs to be altered. Beyond providing intuitive interaction for the performer, the equation of three-dimensional physical gestures with sonic alterations assists the audience in comprehending the electronic instrument. The visual component of live performance is a very expressive tool; if the audience is able to identify the relationships between the performer's gestures and the quality of the sound, they are much more likely to be sympathetic to the musical intent. 4. OPERATION OF THE INSTRUMENT 4.1. Recording a sample The first stage in the operation of the instrument is the procuring of a sound sample to be manipulated. This sound can either be recorded from a live sound source or recalled from a sound file. In the current implementation, a foot pedal is used to control live sampling. Once the pedal is depressed, the audio signal is fed through a partial tracking stage. Using Miller Puckette's "fiddle-" object to identify up to 10 prominent partials in the spectrum, this stage applies a series of bandpass filters to the signal. The spectrum is thus divided into multiple "slices," each focused around a single partial. In addition to this group of processed signals, the original signal is passed unaltered; all of the signals are then stored in one of three arrays of buffer- objects. Using a separate storage buffer for each "slice" of the spectrum affords the performer flexibility in the manipulation of the spectral components. 4.2. Visual Feedback As the sound is being recorded, the average intensity of each frequency band is measured and compared against a cut-off value, which is set in advance by the user. This measurement determines the relevance of a particular partial within the spectrum; whether a partial is sufficiently strong or not is displayed in the Playback area of the screen, indicated by the green highlighted numbers in Fig. 2. During the operation of a playback unit, the performer is given control over whether or not a particular partial will be present, and can use the visual feedback provided by the intensity indicators as an aid. Figure 1. The P5 Glove Controller. Figure 2. The playback stage. 489

Page  00000490 4.3. Playback The performer has a variety of options for playing back the samples collected in a particular array. A trigger message can be sent to all of the buffers simultaneously, so that the each sample appears in the same relative temporal position as when it was recorded. This will in effect re-synthesize the original sound from the array of partials. In order to modify the timbre and focus the listener's attention on particular aspects of the spectrum, the playback of any partial can be disabled; the performer can thus begin to deconstruct the sound by removing specific partials from the spectrum, isolating various components of the sound. Furthermore, the partials can be differentiated from one another through temporal displacement, effected by the adjustment of the "Randomness" control. This value sets the range in milliseconds into which the trigger times for each partial will fall (set to 0 for a unison trigger.) In order for the performer to have adequate time to explore the characteristics of a sound sample, the instrument can employ a looping system, creating a separate loop for each partial. An overlap (cross-fade) value can be set between successive iterations of each sample, giving the impression that the sound is being extended in time. The loop rate can be set to be identical for each partial, or it can vary within a range of values. In addition, silence can be inserted between successive repetitions, set by the "Porosity" value. Through this system of controls, it is thus possible to produce a gradual disintegration of a unified spectrum: the source sound is first resynthesized in imitation of the original sound; particular partials can then be removed and added, partials can begin to drift temporally so they are no longer in sync, and finally spaces can appear between iterations of the partials to aurally separate the elements. The progressions between these stages can be performed in a smooth and overlapping manner, effecting a transition from a single, unified sonic object to a collection of discrete sonic components. Fig. 3 illustrates such a process of disintegration. Each rectangle in the diagram represents the contents of a particular buffer containing a single partial; over time, the Randomness and Porosity values increase, effecting a transition from a unified spectrum to a texture of distinct elements. While this progression would produce a relatively straightforward transformation, the parameters can be applied in a variety of combinations to create a wide range of sonic effects. To retain a connection with the original source sound, the performer is able to mix the original sound with the resynthesized sound at any point, using the crossfade control at the bottom of the Playback unit. For further control over the spectral characteristics of the sound, a standard bandpass filter is appended, allowing control over cutoff frequency and Q value. Finally, the signal is passed through a gain control before being sent to the audio output. 4.4. Glove Mapping Using the matrix interface shown in Fig. 4, incoming values from the P5 glove can be routed to the particular parameters that the performer wishes to control. As well, the performer can set a range of values for each parameter within which they can navigate. In Fig. 4, the glove's X, Y, and Hand controls are mapped to the filter parameters, while the Z dimension controls the cross-fading between the original sound and the resynthesized sound; many other configurations are possible, each creating a unique relationship between the glove's spatial position and the sonic effect. Through experimentation, a performer can find the particular configurations that produce suitable results; these settings can then be saved as a preset, and rapidly recalled during performance with a key command. A wide variety of effects can thus be readily obtained in a live situation without the need to take hold of a mouse or enter numerical values. Figure 4. Glove mapping matrix. As mentioned above, a computer keyboard control is utilized in addition to the P5 glove. While the glove presents an ideal interface for the simultaneous control over continuous parameters (such as frequency or gain values), the keyboard is perfectly suited for sending discrete messages. Key controls allow the performer to change which of the playback units is "active" and thus controllable by the glove (all units will play simultaneously, but only one can be modified by the glove data at a time), and affords control over any of the buttons found on the interface, including partial on/off toggles and a dialogue box for switching between harmonic and inharmonic sound. While all of Spectrum Distinct spectra compn ci s Figure 3. Deconstruction of a spectrum. 490

Page  00000491 the buttons can be controlled with the mouse, the use of the mouse requires a great deal more effort in a performance situation than keyboard controls, which the performer can operate while looking elsewhere. With simple keystrokes, the performer can rapidly change the sound that the glove is manipulating as well as the glove's effect on that sound. Although the user would generally not need to alter the mapping of keys to functions, the ability to do so is provided. 4.5. Creating Complex Textures In certain musical settings, it can be desirable to generate complex textures that combine multiple simultaneous processes, each using a different sound source. Since such textures may be beyond the performer's ability to control with the gestures of one hand, the Spectral Instrument allows the user to program and trigger extended processes that engage any of the playback units, using the Max/MSP "qlist" object. A performer could thus establish a process that loads a particular sample into the first playback unit, begins a looping texture, and then removes specific partials one at a time over a period of three minutes, while increasing the porosity value. A process such as this could be triggered by a single keystroke, creating a backdrop over which the performer could improvise using the primary playback unit. At any time during a qlist process, the performer can interrupt the automatic sequence and assume manual control over that playback unit. In addition, qlists can control playback modules external to the Spectral Instrument, such as a soundfile player or an FM synthesizer. By layering multiple streams of sonic processes, the performer is able to create interesting and varied textures while retaining intuitive and expressive gestural control over a primary sound stream. 5. ARTISTIC IMPLICATIONS The application of the Spectral Instrument in the piece from that which could for voice and electronics (David Litke, 2007) draws an analogy between the decomposition and re-composition of an acoustic spectrum and those same processes in the musical and textual semantic content. When isolated, individual partials do not engender a semiotic link with a specific mode of sound production as they do when combined into a spectrum, just as individual phonemes do not typically acquire representational meaning until they are combined to form words and sentences. In the absence of these symbolic connections, the qualitative aspects of the sounds are emphasized. Once the elements are combined, however, the listener's attention shifts from the acoustic phenomena towards the sound's representational significance. In the introduction to this work, the timbre of the voice is reconstructed by successively adding partials, creating a transition from the perception of discrete pitches to a unified overtone structure. Similarly, the vocal line begins by presenting a collection of vowel and consonant sounds, and progressively assembles these materials into words. As the piece progresses, other timbres (such as a violin and a bell) are re-synthesized, effecting transitions from the perception of acousmatic partials to timbres with an identifiable source. Following the climax of the work, the complex sounds are again pulled apart, dissolving the timbral identities into a variety of textures. In tracing these transitions, the music thus explores the liminal regions of perception, as discussed by Grisey [2]; there are moments during the transitions at which the listener is unsure whether he or she is hearing multiple discrete pitches, or whether the sound constitutes a single spectrum with an identifiable source. Since one of the guiding metaphors in the operation of the instrument is of taking hold of the sound of a voice or instrument, and then manipulating that sound in a "sonic space," the use of a glove controller has added dramatic effect. As the instrument is primarily intended to be used in conjunction with acoustic sound sources (i.e. other performers playing physical instruments), the visual impact of the performance will clearly indicate the relationship between the Spectral Instrument and the other players: while the other players physically send vibrations through the air, the Spectral Instrument player's hand appears to act only on the medium, manipulating not a solid object but the very sound as it travels. 6. FURTHER WORK As mentioned above, the data obtained by the Spectral Instrument can be exported to other modules; one area of further research involves finding artistically viable methods of interpreting and applying this data. Furthermore, it remains to be explored how other types of continuous controllers, such as motion capturing systems for dancers, could be applied in place of the P5 glove. ACKNOWLEDGMENTS I would like to thank Dr. Bob Pritchard for his assistance on this project, which is associated with the GRASSP project and supported by the SSHRC foundation (grant number 848-2003-0147). REFERENCES [1] Grisey, G. Partiels. pour 18 musiciens. Ricordi, 1976. [2] Grisey, G. "Structuration des timbres dans la musique instrumentale", in Le timbre, metaphore pour la composition. Bourgois, 1991. [3] Zicarelli, D. Max 4.2 Reference Manual. Cycling '74, www.cycling74.com. 2003. 491