Page  00000001 Software Tools for Musical Expression Roberto Bresin and Anders Friberg Department of Speech, Music and Hearing (TMH) - Royal Institute of Technology (KTH) Drottning Kristinas vag 31, SE-10044 Stockholm, Sweden {roberto, andersf}@ speech.kth.se http://www.speech.kth.se/music ABSTRACT In this article software tools that model principles used in music performance are presented. They all add expressive variations to a score-based representation of the music. The two main tools are Director Musices, a Lisp program, and Japer, a Java applet. Some possible applications are illustrated: music production, teaching of music performance, and performance of mobile phone ringing tones. INTRODUCTION In the long-term research project on music performance at the Speech Music and Hearing Department (TMH) at KTH, Stockholm, several tools have been developed. These tools are designed to add performance nuances to existing scores, similarly to how skilled players interpret music. The translation of the nominal score to a performance is done either by a set of rules (e.g. Friberg, 1991; Friberg and Sundberg, 1999; Friberg et al. 1998; Friberg 1995), by neural networks (Bresin, 1998; Friberg et al. 1998), or by a combination. They operate on tone parameters such as duration, loudness or vibrato rate. The tools are written in the Java and Lisp programming languages. Input and output data can be files either in MIDI or in custom formats, as well as real-time MIDI commands. The tools have been used to control software for synthesis of the singing voice, software for virtual dance animation and for commercial MIDI software and hardware. DIRECTOR MUSICES Director Musices (DM) is the Lisp program that has been used for the development of the KTH performance rules (Friberg et al., forthcoming). Many rules for marking the musical structure, such as phrasing, intonation, timing, melodic emphasis and ensemble synchronization, are implemented. Recently, emotional coloring (Bresin & Friberg, 2000) as well as random variations has been modeled and included in a complete model for music performance: the GERM model (Juslin et al., 1999). DM allows for arbitrar ily complex polyphonic music (consisting of elements with a duration and pitch) with continuous variation of all performance variables. The kernel in DM 2.0 has been rewritten allowing more complex structures to be processed in a hierarchical way. One new feature is the possibility to have vocal tracks with rules for pronunciation. The user interface includes graphs of all performance variables as well as music notation. Rule palettes are used for storing rule setups including rule parameter values. DM is written in common lisp and is available as a stand-alone application both for Macintosh (PowerPC and 68k) and PC (Windows 95, 98 and NT). It is easy for the user to define new rules with the provided rule macros and variable access functions. In addition, all the resources in standard common lisp are available in a rule definition. A typical DM working session is presented in Figure 1. JAPER: JAVA PERFORMER Japer, JAva PERformer, is a Java applet implementing a sub-set of the rules included in DM (Bresin, 1999), see Figure 2. Japer takes advantage of the MidiShare operating system developed at GRAME (Orlarey and Lequay, 1989; Fober, 1994; Orlarey, 1994). We chose the Java-MidiShare combination mainly for the following reasons: (1) complete portability between different operating systems (Mac OS, Windows, Linux), (2) possibility of running an applet in a WWW browser and thus of interaction with existing music database and other Internet-based services, (3) small dimensions of the Java code allowing short downloading time, (4) Midishare allows a Java applet to exchange real-time Midi messages with any Midi device attached to the client machine. Other JAVA applets that have been developed are Pann and Jalisper. Pann, based on the same structure of Japer, is a program implementing artificial neural networks for detecting punctuation marks in music scores (Friberg et al., 1998). Jalisper is a Java-Lisp performance system, where a client GUI is developed in Java and the performance rules are implemented in a Lisp program running as server.

Page  00000002 501Y" Ll y 40 30 2 201 10 2.3.'4 6 7 8 10.11 12 14 -1......... -20 \ *--r----- I Fl d I I I I I I I I d - Ap d9;00 lp d io. io -40;tj -40 Ig -40;5 Figure 1. Screen shot from Director Musices. The rule palette window contains a selection of rules and its parameters and defines a particular performance style The resulting variation in interonset duration is shown in the bottom window. Figure 2. Screen shot of the Japer Java applet for automatic music performance. In this picture Japer is playing a performed version of a melody by Bellman.

Page  00000003 APPLICATIONS Music production A large part of contemporary music is produced directly on the computer. Using existing commercial software, such as sequencers and music notation software, it is hard to shape the expressive content of the performance. Here the performance models can reinsert the important, usually intuitive, but elaborated, knowledge of music performance and interpretation into the music production of today. Teaching music performance Another important application is in the music education. The scientific progress as well as possibilities to analyse data from performances define a new autonomous area in the field of music analysis by using new software tools (Friberg and Battel, forthcoming). Studies show that relatively little time in music education is spend on interpretative aspects of performance (Persson, Pratt, & Robson, 1992). The new tools described can help to reintroduce this important aspect of music performance into the classroom. The models for music performance can be applied to a given piece with separate control of each performance aspect using different amounts and parameters, illustrated in graphs, and listened to. This has a number of advantages. For example, the possibility of exaggerating the effects can be used so that anybody, regardless of musical training, can hear the difference and can focus on a particular aspect of performance. It is also useful to compare a model performance with the actual performances by the student and discuss similarities and differences. Following these ideas, Battel introduced a new teaching method at the Venice Music Conservatory (Friberg & Battel, forthcoming). Also interesting is the comparison between a natural performance and performances with particular expressive intentions (Battel and Fimbianti, 1998; De Poli et al., 1998; Bresin & Friberg, 2000). The knowledge of how the performance parameters vary for different intentions gives a more nuanced picture than similar studies which focuses mainly on the structural communication of the score through performance deviations produced by the player. Swing lab Timing is probably the most discussed aspect in jazz performance. The swing ratio, i.e. the duration ratio of the long-short pattern of the consecutive eighth notes, is important for the swing feel of a melodic/rhythmic line (Friberg and Sundstrim, 1997). Another important issue is the concept of playing before/behind the beat, that is, the relation to the other members of an ensemble. Both aspects can be studied in the set-up shown in Figure 3. The student's own timing as well as the timing in recorded performances can be analysed in a normal spectrogram. A model for jazz ensemble performance in DM can be used to produce examples as well as serving as a comping pattern that the student can play along with and at the same time analyse. Ringing tones in mobile phones In the "lo-fi" area of music performance such as in games or ringing tones, the music is often represented by some score notation similar to a MIDI file. Hence, there are many possibilities in this field to apply the models for music performance. For instance, a given melody in a game can be played in a sad or happy way depending on a particular user action (Bresin & Friberg, 2000). The ringing melodies in mobile phones is something that often is irritating, especially for musicians. Here, better performances would significantly increase the pleasantness of the signal, although the quality of the sound also needs to be improved. Timing parameters Figure 3. The swing lab set-up. The model can generate performances by a jazz ensemble consisting of drums, bass and a soloist where both swing ratios and timing within the ensemble can be varied. By omitting one of the generated instruments, a student can play along with the model and directly in real time see the resulting timing in a spectrogram.

Page  00000004 LINKS KTH performance rules description: http://www.speech.kth.se/music/performance/ Director Musices (Windows & Mac OS): http://www.speech.kth.se/music/performance/download JAPER: http://www.speech.kth.se/-roberto/japer PANN: http://www.speech.kth.se/-roberto/pann REFERENCES Battel G. U., R. Fimbianti. 1998. "How communicate expressive intensione in piano performance." In A. Argentini and C. Mirolo (eds) XII Colloquium on Musical Informatics: Expression and Performance Analysis - II, pp. 67-70. Bresin, R. 1998. "Artificial Neural Networks Based Models For Automatic Performance of Musical Scores." Journal of New Music Research 27(3): 239-270. Bresin, R. 1999. "JAPER and PANN: two JAVA applets for music performance." Included in the CD-ROM "MidiShare: Operating System for Musical Applications", National Center of Contemporary Music - GRAME, Lyon, http://www.grame.fr Bresin, R., and A. Friberg. 2000. "Rule-based emotional colouring of music performance." Proceedings of the ICMC 2000, Berlin, in this publication De Poli G., A. Rodh, and A. Vidolin. 1998. "Note-by-Note Analysis of the influence of Expressive Intentions and Musical Structure in Violin performance". Journal of New Music Research 27(3): 293-321. Friberg, A. 1991. "Generative Rules for Music Performance: A Formal Description of a Rule System." Computer Music Journal 15(2): 56-71. Friberg, A. 1995. "Matching the rule parameters of Phrase arch to performances of "Trdiumerei": A preliminary study." in A. Friberg and J. Sundberg (eds.), Proceedings of the KTH symposium on Grammars for music performance May 27,1995, pp. 37-44. Friberg, A., and G. U. Battel. Forthcoming. "Structural Communication: Timing and Dynamics." In R. Parncutt and Gary McPherson (Eds.) Science and Psychology of Music Performance. Friberg, A., R. Bresin, L. Fryden, and J. Sundberg. 1998. "Musical punctuation on the microlevel: Automatic identification and performance of small melodic units." Journal of New Music Research 27(3): 271-292. Friberg, A, V. Colombo, L. Fryd6n, and J. Sundberg. Forthcoming. "Generating Musical Performances with Director Musices." Computer Music Journal. Friberg, A., and J. Sundberg. 1999. "Does music performance allude to locomotion? A model of final ritardandi derived from measurements of stopping runners." Journal of the Acoustical Society of America, 105(3): 1469-1484. Friberg, A., and A. Sundstr6m. 1997. "Preferred swing ratio in jazz as a function of tempo." Speech Music and Hearing Quarterly Progress and Status Report, 4/1997: 19-28. Juslin, P.N., A. Friberg, and R. Bresin. 1999. "Towards a Computational Model of Performance Expression: The GERM Model." Paper presented at the Meeting of the Society for Music Perception and Cognition (SMPC'99), Evanston, USA Orlarey Y. 1994. "Hierarchical Real Time Interapplication Communications." Proceedings of the ICMC 1991, ICMA, San Francisco, pp. 408-415. Orlay Y., and H. Lequay. 1989. "MidiShare: a Real Time multi-task software for Midi applications." Proceedings of the ICMC 1989, ICMA, San Francisco, pp. 234-237. Persson, R. S., G. Pratt, and C. Robson. 1992. "Motivational and influential components of musical performance: A qualitative analysis." European Journal for High Ability 3: 206-217.