Page  179 ï~~Real-Time Sound Hybridization Nicola Bernardini Alvise Vidolin Centro TEMPO REALE Centro di Ricerca, Produzione e Didattica Musicale Villa Strozzi Via Pisana 77 50143 Firenze Tel.++39(0)55/717270 FAX ++39(0)55/717712 E-mail: {nicb,vidolin} Abstract This paper presents a case study on real-time sound hybrids connected to a specific musical application, namely the orchestra, choir and live-electronics work by Luciano Berio Ofanim. In Ofanim, some passages ask for specific real-time hybrids produced by the combination of two instruments among a clarinet, a trombone and a boy's voice. These hybrids are the result of a sound produced by a 'real' instrument and a pitch-locked synthesized sound representing the 'other' (virtual) instrument. 1. Introduction In 1991, during the realization of the fourth version of Ofanim Luciano Berio asked us to provide some form of hybridization between the singing voice of a boy, a clarinet and a trombone. The effect Berio desired to achieve was to create a new timbre space in which the continous transition between any one of the three sounds with each other would be possible at any time. The very same concept was somewhat illdefined and it was very hard, for us and the composer to clarify the phenomenological nature of what was to happen. Furthermore, as with the rest of Ofanim, this hybridization was to happen in real-time, and this fact posed problems that went beyond the expected ones, that is, the technological diffculties posed by any digital real-time process. We decided therefore to begin in a very practical way, carrying out several experiments in different contexts to try to define what is timbral hybridization and how does one go about to achieve it in real time. In the meantime the interest in timbral hybrids has grown both in the research and in the industrial fields, helping us out and suggesting other solutions to achieve our goals (Serra et al., Aikin). At any rate, though we have gone a long way since our first experiments there is still space for further research and improvement. 2. Problems to solve Before describing the different solutions we found to real-time sound hybridization we will try to make a list of the major problems that we had to face. Real-time, first of all. The real-time performance of sound hybridization implies that the "real" instrument must be "hybridized" with another instrument which does not exist in physical reality but is only perceived by the listeners through an audio system. This latter instrtument, which for the sake of convenience we will ICMC PROCEEDINGS 199517 179

Page  180 ï~~call the "virtual" instrument, must be linked to the "real" instrument satisfying the contraints of spectral fusion in order to perceive the resulting effect as a unique product of a single source. This is easier said that done, since the ear is particularly finnicky about spectral fusion. Also, the constraint of real-time obliged us to avoid machine-consuming methods such as vocoding: the vocoder which we could have built in real time (using some 10 filters or so) was just too crude to be of any use. The connection between the ear and the eye produces a side effect which creates the second problem: when we are listening to an instrument we tend to continue to perceive that instrument even though the sound has changed to a great extent. It is a sort of "perceptual hysteresis" which is probably created by the global amount of audiovisual information conveyed by the playing of a musical instrument. Incidentally, we discovered after a while that a "perfect" electronic interpolation between two real-world timbres is not so interesting after all: as a matter of fact it is easily realized with standard orchestral means; in real-time timbre hybridization the characteristics of the "virtual instrument" must be downright exaggerated in order to overcome our tendency to stick perceptually to the "real instrument". This is why we ended up spending less time in perfecting "virtual instrument" simulations than on the transition mechanisms themselves and on amplification of the main characteristics of a given instrument. The third problem, perhaps the most difficult to tackle, concerns the production of the sound of the "virtual" instrument. This instrument must be realized electronically and must be able to be both an hybrid of the two sounds at the extreme ends of the effect and a plausible simulation of the target instrument. Finally, there is also a more musical problem in creating sound hybrids: the music for the "hybridized" instrument must be written keeping in mind the target "virtual" instrument and the player her/himself must follow some form of mimetic behaviour in order to achieve the desired effect. This may sound perhaps self-evident to most musicians, but such "mimetic" attitude both in writing and playing is not easy to achieve in all details, because it is not sufficient to just "write for the virtual instrument". Some details, like the addition of portamento to a melody played by a "clarinet-to-voice" hybrid, must be produced at the processing level in order to reach some degree of plausibility. 3. Some solutions In general, the set up to produce real-time sound hybrids is the following: Real Virtual Instrument Instrument Envelope Pitch Follower Detector RT Control Crossfade Figure 1. Real-time sound hybrids environment With few modifications to this generic path, we have adopted, up to now, several solutions which in general perform with some success on a specific hybridization pattern. We will list here the partially successeful ones devoting a specific sub-paragraph to each one: 1. Pitch and envelope followed interpolation (mixing) of the "real instrument" with a sampled copy of the "virtual instrument" 2. Pitch synchronous interpolation and envelope following of the "real instrument" with a synthesized form of the "virtual instrument"; we have tried two different forms of synthesis with this system: 180 1801CMC P RO C EE D I N G S 1995

Page  181 ï~~" Frequency modulation " Ring Modulation 3. Non-linear distorsion of the original signal 3.1 Pitch followed mixing This has been our very first attempt to produce timbre interpolation. The idea is quite simple and easy to set-up; the surprising part of it is that some results are actually quite good and efficient in a concert situation. All is needed is a good pitch-detector, a digitally controlled mixer and a sampler. The design of mixing interpolation tables can actually make a big difference in the results. This algorithm as proven to be fairly successeful, in our case, to "hybridize" a clarinet into a singing voice. The big problem with this design is obvious: there is no real "hybrid" at any middle point of the interpolation; any intermediate result is almost always felt as a mixing of two signals because it is the mixing of two signals, even though the spectral fusion obtained with pitch detection and envelope following may mask the crudeness of the patch. Thus, this algorithm may work sufficiently well at the extremes but is very poor in the middle. 3.2 Pitch synchronous Frequency Modulation The next idea was to develop a tighter form of pitch detection (that is, continuous pitch detection instead of note onset MIDI controls) and to use the envelope of the original signal to control some significant timbral parameters of the "virtual instrument". The setup became more complicated and involved the use of a realtime processing hardware. We have used the MARS developed by LRIS-Bontempi (Andrenacci et al.) and Emmanuel Favreau' developed to this end a zero-crossing pitch detector which could actually follow the period of the original signal with a delay of a cycle. Then we built some frequency-modulation simulation of the various instruments controlling the carrier and modulation frequencies of the signal with the pitch detector and the modulation indexes with external controls and with the appropriately rescaled values coming from the envelope follower. In this way, that is not only by mixing the two signals but acting upon the modulation indexes it was possible to achieve a higher degree of hybridization. The problems that we encountered with this model are the ones connected to all synthesis by frequency modulation: it is a well known fact that frequency modulation is a syntesis technique that suits some form of timbre modulation better than others. In our case, the simulated trombone reached some form of plausibility while the clarinet and the voice did not. 3.3 Pitch synchronous Ring Modulation The ring modulation design uses the pitch detector and the envelope follower to control a sine-wave oscillator which is then multiplied by the original signal. The relationship between the carrier signal (the sine wave) and the modulating signal (the "real instrument") must be set at a 1:2 ratio so that the result comes out at the original pitch. This technique gave surprising results for all instruments, even though a true simulation was hardly reachable with it. 3.4 Non-linear Distorsion The non-linear distorsion setup was rather different: a filtered-out (almost to the fundamental) version of the "real instrument" was used to drive the index of a table in which different Chebychev polynomials were experimented in order to simulate the various "virtual instruments". This model proved to be a fairly good model to create smooth transitions, but in the end we had to abandon 1. at the tine with IRIS-Bontempi, now with the GRM in Paris I C M C P R OCE E D I N G S 199518 181

Page  182 ï~~it because the target instrument simulations were too poor. This provided, in fact, a specific case of lack of agogic details of the "virtual instrument" mentioned in the introduction: since the synthetic instrument is generated through a distorsion of the real one, all the articulation information is tightly related to the latter and any attempt to simulate some other instrument fails. 4. Conclusions The limits of the experiments we made suggested the directions we ought to develop in the future. They can be summed up as follows: 1. we have not found a single technique that can work as a general model; we were forced to adopt different setups and methods in order to achieve something mildly close to what was required; 2. as Abraham Moles stated many years ago (Moles), the various portions of the time domain of a sound are not all equal in terms of information content: the attack, for one, is more important than any other portion of sound in determining timbre. The sustained portion, in turn, can be the smoother vehicle to timbral interpolation. Briefly, it is necessary to develop some form of "morphological interpolation" (somewhat dissimilar, however, to what is called "morphing" in some recent industrial instrument; cf. Aikin). We believe that by treating each portion of the sound separately it is possible to achieve a wider palette of timbral hybrids. Last but not least, we would like to acknowledge all the help we have received from different sides. First of all Luciano Berio, who has has given us a lot of musical feedback on the different experiments. Many thanks are deserved by the entire crew that developed the MARS Workstation at IRIS in Rome, and in particular to Giuseppe DiGiugno and Emmanuel Favreau who participated actively to many of our experiments, both in the lab and in concert. Many thanks go to Curtis Roads too, who has provided new insights in the possibilities of hybridization through granular synthesis (Roads). 5. References Aildn, J. E-mu Morpheus, in Keyboard Magazine, March 1994, pp.96-110 Andrenacci, P., Favreau, E., Larosa, N., Prestigiacomo, A., Rosati, C. & Sapir, S. MARS: RT2OMIEDIT20 - Development Tools and Graphical User Interface for a Sound Generation Board, in Proceedings of the 1992 International Computer Music Conference, San Francisco: Computer Music Association, pp.344-347 Moles, A. Les Musiques Experimentales, 1960 Roads, C. Sound Composition with Pulsars, unpublished paper Serra, M-H., Rubine, D. & Dannenberg, R. Analysis and Synthesis of Tones by Spectral Interpolation, in Journal of Audio Engineering Society, vol.38 n.3, March 1990, pp.111-128 182 2ICMC PROCEEDINGS 1995 1995