Page  190 ï~~A Musical Application of Real-time Granular Sampling Using the IRCAM Signal Processing Workstation Cort Lippe IRCAM, 31 rue St-Merri, Paris, 75004, France email: lippe@ircam.fr 1 Introduction Interest in granular synthesis, along with compositional strategies for exploring this technique, have been presented by various composers [Xenakis, 1971], [Roads, 1978]. More recently, important compositional and technical results have been presented in the domain of real-time granular sampling [Truax, 1987], which has proven to be a powerful technique for timbral transformation of sampled sounds in real time. This paper discusses essential differences between granular synthesis and granular sampling techniques, presents work by the author using "time-stretching" techniques [Jones & Parks, 1988], and describes a musical application using the IRCAM Signal Processing Workstation (ISPW) [Lindemann, Starkier & Dechelle, 1990] in which granular sampling is controlled via nonlinear processes in a real-time compositional environment [Truax, 1988], [Washka & Kurepa, 1989], [Di Scipio, 1993]. Compositions by the author for instruments and live ISPW are presented which make extensive use of "expressive" control of granular techniques via the detection and tracking of musical parameters of live instruments in real time [Lippe & Puckette, 1992], [Wessel, Bristow & Settel, 1992]. Finally, a user interface, developed by the author using the program Max [Puckette, 1988, 1991] running on the ISPW, is described. 2 Description of Granular Techniques A simple description of a granular synthesis model includes the following constants: a sinusoidal waveform, a bell-shaped amplitude envelope with a duration of 20 milliseconds, and an overlap time of 5 milliseconds between successive grains. Grains of sound, produced at a high rate of speed, are usually overlapped with neighboring grains in order to produce a certain density and continuity of sound. Pitch, maximum amplitude of individual grains, and grain density (rate of grain production and overlap of successive grains) may be considered as compositional variables in this model. 2.1 Granular Sampling The technique of granular sampling involves the application of the above-described technique whereby the waveform used in granular synthesis is replaced by a small chunk of sampled sound. Thus, for each grain, the onset time into a sampled sound becomes a compositional variable, along with the pitch, amplitude, and grain density. In a more detailed model of granular sound production, the waveform (or sampled sound), envelope description, grain duration, spatial location of each grain, etc., may also be considered as compositional variables. 2.2 Granular Parameters With this large palette of available parameters, it is clear that an immense quantity of data may be required in choosing individual values for each grain of sound. Historically, compositional algorithms have often been employed to automate these choices. The practicality and necessity of automating control of granular parameters was obvious to Xenakis, who, prior to working with granular techniques in electronic music, had already explored similar problems in the instrumental domain during the 1950's with works such as Metastaseis, in which he employed techniques akin to a kind of "granular" conception of instrumental music. 3 Compositional Implications of Granular Sampling While granular synthesis and granular sampling are variants of the same technique, their musical essences lie at opposite poles of the electronic music paradigm. One is immediately confronted, historically speaking, with the two main categories of electronic music: granular synthesis is elektronische Musik, making use of purely synthetic sounds, while granular sampling is part of the world of musique concrete in which recorded sounds are manipulated and transformed. As the Canadian composer, Jean Pich6, has suggested, granular sampling is an "input dependent" technique. Thus, using granular techniques on sampled sounds offers an obvious level of musical implication which does not exist in granular synthesis: one is acting on and transforming a preexisting sound object. 3.1 Grain Order As mentioned above, in granular synthesis the parameters most often controlled algorithmically are the pitch, amplitude, and density of grains. While the ordering of grains in a coordinate space is calculated, giving some sort of density distribution, the concept of grains in an ordinal sense remains 3B.1 190 ICMC Proceedings 1993

Page  191 ï~~somewhat abstract. The results of arbitrarily different orderings yield sounding variants which, although possibly very different in nature, remain abstract synthetic sounds. Since the synthetic waveform used in granular synthesis is replaced by a small portion of a stored sampled sound in granular sampling, an additional parameter exists: onset time into the stored sound. This additional parameter can be of primary importance in granular sampling. No longer a kind of "commutative" or arbitrary parameter, grain order may have important consequences, creating an implicit hierarchy of parameters. Using spoken speech as a sampled sound, if onset times descend in an ordinal fashion from high to low, while density distributions of all other parameters are randomly calculated, the sounding result will always be recognized as spoken speech played backwards even though variants may sound quite different. Furthermore, one of the principle musically interesting characteristics of granular sampling is the ability to "deconstruct" sounds via the manipulation of onset times, moving between the boundaries of recognizability and non-recognizability on a continuum. 4 An ISPW User Interface for Granular Sampling Using Max on the ISPW, I constructed an interface for controlling granular sampling in real time. All the parameters mentioned above, including: onset time into the sampled sound, pitch, envelope description, maximum amplitude, grain duration, rate of grain production, overlap of grains, and spatial location of each grain are all controllable in real time for each grain that is calculated. Parameter settings can be given using sliders and number boxes, and parameters can be changed independently over time by automating control processes. Max also allows for real time switching from one sampled sound to another, either by reading elsewhere in memory, loading soundfiles from disk, or sampling anew (all of which can be done while the granular reading continues to take place). Independent granular sampling tasks can run at the same time. Since the point of departure for this work in granular sampling grew out of experimentation with "timestretching" (see below) of sampled sounds, each task originally produced a single stream of grains. Multiple, simultaneous grain attacks were a later development, the number of tasks and the number of overlapping grains within a task being limited by real-time constraints. A recent addition to the system allows for real-time mixing and sampling of the granular output of simultaneous tasks, which then may be reused as stored samples for other granular sampling tasks. This "recursive" aspect offers exponential increases in densities, and a musically "reflexive" dimension, namely, the ability to recall earlier musical material in a real-time context. 5 Time-Stretching of Sampled Sounds Time-stretching of sampled sounds has been studied for altering speech signals for quite some time. Compositionally, this technique offers the possibility to separately control pitch and time in sample playback. Slowing down a sound without changing its pitch, or changing pitch without changing playback speed, have interested composers since the early days of electronic music. In the late 1940's, D. Gabor developed a type of time-stretching which is essentially a form of granular sampling. His technique was commercialized as the Springer taperecorder, which gave composers an analog tool for experimentation [Roads, 1991]. A simple digital model for granular time-stretching makes use of a small number of voices (or read pointers) which read a stored sound in the following way: each voice produces a grain with a duration of 50 milliseconds, with a 15 millisecond overlap between successive voices. This is repeated cyclically on a continuous basis, while a pitch is specified for each grain and a "precession" rate is defined. (Note the similarity to a typical harmonizer algorithm.) 5.1 Precession Rate The precession rate can be described as a sliding window which moves or advances a group of continuously cycling voices as they read through a sampled sound at a certain rate of speed. Since the precession rate does not transpose the original sound, a slowly advancing precession rate can give a slowed-down playback at the original pitch if the grains of sound are left untransposed; or the precession rate can be kept at the original playback speed while the pitch of successive grains can be altered. With this technique, one can freeze the reading process in a single region by giving a precession time of zero, freely move forwards or backwards in a linear fashion, repeat short sections of the stored sound, and constantly vary the playback time and/or pitch independently. (Note that, on a local level, this technique does not change the sequential order of onset times into the sampled sound.) After exploring the above-mentioned possibilities of time-stretching in my compositions Music for Harp and Tape, Music for Guitar and Tape, and the tape piece Paraptra; in which stored samples of harp and guitar sounds were transformed, it was a simple step to produce granular sampling by modifying timestretching techniques in order to control the onset times into stored sample in a non-sequential fashion. 6 Initial Experiments with Granular Sampling My initial experiments with granular sampling were extremely simple: The auditory result of randomly choosing onset times into a stored sound, while producing a single stra of grains at the original pitch of the sound, is fairly statistical. Using samples of ICMC Proceedings 1993 191 36.1

Page  192 ï~~instrumental music, one has the impression with certain phrases that, for example, an entire 10-second phrase is sounding simultaneously. This is not surprising, since any onset is just as possible as any other, and, since, in using 20-millisecond grain durations with overlaps of 5 milliseconds between successive grains, more than 60 grains are produced each second. (Increasing the overlap time between grains will greatly increase the density of grains per second.) One sampled clarinet phrase, in particular, made up of approximately 5 seconds of rapid short notes and then a 5-second held note, was noteworthy because of the omnipresence of the long note in the statistical sound mass. It was immediately obvious that the musical content of the stored sounds being operated on was not a trivial aspect of the procedure, and that mapping algorithmic calculations onto a stored sound might produce more or less successful results if the musical content of the sound was taken intoaccount. 7 Nonlinear Control of Granular Parameters A first attempt at controlling granular sampling using nonlinear mapping was simply to choose grains statistically within defined "tendency masks" (constantly moving windows with varying sizes in which grains are chosen). For instance, a window with the size of a single grain moving forward in a sound which expands to the size of a full 10 second stored sample over a specified time produces a sound which begins untransformed and, over time, becomes a statistical sound mass. These tendency masks of constantly moving window sizes and window locations can be used to read through sounds quite freely in a kind of statistical "scrubbing" fashion, creating more or less recognizable playback of the original sounds with a rich amount of timbral variation. Random walks through the sound can be calculated and combined with control over the numerous other parameters available: pitch, amplitude, choice of stored sample, envelope description, grain duration, rate of grain production, overlap of grains, and spatial location of each grain, giving one a vast amount of transformational flexibility. My composition Music for Clarinet and ISPW employs variants of time-stretching and granular sampling, making use of tendency masks to control virtually all the parameters of granular sampling. In a work in progress, algorithms have been tried for controlling different parameters using chaotic equations. These algorithms can be predictively and easily controlled, enabling smooth transitions from the seemingly random towards stability, makes them quite attractive. 8 Mapping Performer Expression to Control Parameters Due to the large number of parameters and the much larger number of values needed for each parameter in granular techniques, it is obvious that algorithmic mappings can be extremely useful, if not necessary. Several of the pieces mentioned above involve the use of live performers. Since the ISPW offers tools for real-time audio signal analysis of acoustic instruments for the extraction of musical parameters, another level of control over the granular sampling comes directly from the performers, giving musicians a degree of expressive control over the electronic transformations. In Music for Clarinet and ISPW, the sampled sounds used for granular sampling are taken directly from the performed score, either sampled on-the-fly, or prerecorded and loaded into memory during performance. Continuous pitch and amplitude tracking of a performance offers musically relevant data which can be used to control aspects of an electronic score, and perceptually create coherence between the instrument and electronics. In the clarinet piece, continuous pitch data taken from the clarinet is often used to control the pitch of grains, and continuous amplitude data controls the windowing of the tendency masks of certain parameters. In a work in progress, additional control of parameters is being attempted via spectral analysis, thus allowing for timbral control of the sampling by way of instrumental color changes. (See figure 1 below.) deis disdew and canofspc aI mg. piatch d ily, etc figu i. Mapping pernor expreuion 9 Conclusion Granular sampling is a powerful tool for transforming sampled sounds. Control of granular sampling via nonlinear processes in a real-time compositional context, and via continuous control signals made available by the detection and tracking of musical parameters of live instruments in real time, offers composers and performers a rich palette of possibilities. In addition, sampling of the output of a performer in a real time environment, while allowing the performer a certain degree of control over the 3B.1 192 ICMC Proceedings 1993

Page  193 ï~~granular sampling of this same material, can ultimately provide an instrumentalist with a high degree of intimate expressive control over an electronic score. 10 Acknowledgements I would like to thank Miller Puckette, Jean Pich6, and Agostino Di Scipio for their invaluable technical and musical insights. References [Di Scipio, 1993] A. Di Scipio. "Composing with Granular Synthesis of Sound in the Interactive Computer Music System." In D. Smalley and N. Zahler, eds. Proceedings of the Fourth Biennial Arts & Technology Symposium. New London: Center for Arts & Technology at Connecticut College, 1990. [Jones & Parks, 1988] D. Jones and T. Parks. "Generation and Combination of Grains for Music Synthesis." Computer Music Journal 12(2):27 - 34, 1988. [Lindemann, Starkier & Dechelle, 1990] E. Lindemann, M. Starkier, and F. Dechelle. "The IRCAM Musical Workstation: Hardware Overview and Signal Processing Features." In S. Arnold and G. Hair, eds. Proceedings of the 1990 International Computer Music Conference. San Francisco: International Computer Music Association, 1990. [Lippe & Puckette, 1992] C. Lippe and M. Puckette. "Musical Performance Using the IRCAM Workstation." In B. Alphonce and B. Pennycook, eds. Proceedings of the 1991 International Computer Music Conference. San Francisco: International Computer Music Association, 1991. [Puckette, 1988] M. Puckette. "The Patcher." In C. Lischka and J. Fritsch, eds. Proceedings of the 1988 International Computer Music Conference. San Francisco: International Computer Music Association, 1988. [Puckette, 1991] M. Puckette. "Combining Event and Signal Processing in the Max Graphical Programming Environment." Computer Music Journal 15(3):68 - 77, 1991. [Roads, 1978] C. Roads. "Automated Granular Synthesis of Sound." Computer Music Journal 2(2):61 - 62, 1978. [Roads, 1991] C. Roads. "Asynchronous Granular Synthesis." In G. De Poli, A. Piccialli, and C. Roads, eds. Representations of Musical Signals. Cambridge: The MIT Press, 1991. [Truax, 1987] B. Truax. "Real-Time Granulation of Sampled Sound with the DMX-1000." In 1. Beauchamp, ed. Proceedings of the 1987 International Computer Music Conference. San Francisco: International Computer Music Association, 1987. [Truax, 1988] B. Truax. "Real-Time Granular Synthesis with a Digital Signal Processor." Computer Music Journal 12(2):14 - 26, 1988. [Washka & Kurepa, 1989] R. Waschka and A. Kurepa. "Using Fractals in Timbre Construction: An Exploratory Study." Proceedings of the 1989 International Computer Music Conference. San Francisco: International Computer Music Association, 1989. [Wessel, Bristow & Settel, 1992] D. Wessel, D. Bristow and Z. Settel. "Control of Phrasing and Articulation in Synthesis." Proceedings of the 1987 International Computer Music Conference. San Francisco: International Computer Music Association, 1987. [Xenakis, 1971] 1. Xenakis. Formalized Music. Bloomington: Indiana University Press. (Pendragon, 1991) 1971. ICMC Proceedings 1993 193 3B:1