Page  00000001 Articulation Rules For Automatic Music Performance Roberto Bresin Department of Speech, Music and Hearing Royal Institute of Technology, KTH, Stockholm email: roberto.bresin@speech.kth.se Abstract Rules for legato, staccato, and repeated notes articulation in automatic piano performance are presented. These rules were obtained applying the analysis-by-measurement method to performances of Mozart piano sonatas on MIDIfied acoustic grand pianos. The effects produced by the rules vary with tempo indications, with expressive intentions and with musical context. The articulation rules are part of the Director Musices system for automatic music performance. It has been demonstrated that proper setting of the articulation rules enables more realistic automatic performances that can also differ in emotional expression. These rules are also being applied in control models for sound synthesis algorithms. 1 Introduction Articulation is one of the most important cues in music performance, since it allows control of the way in which two consecutively played notes are acoustically connected, and contributes to the character of the performance. Much effort is spent on the study of articulation in musical training. In the past, few researchers have paid attention to the analysis of articulation in piano performance. This could be due to two main reasons. First, it is difficult to detect the instant when a key is released and when the associated sound has passed the threshold of audibility. Second, a precise measurement of the mechanical movements of piano keys and hammers is possible only in commercial MIDIfied pianos like Disklavier and B6sendorfer, or in pianos provided with various sensors. Examples of the latter are photocells, as used by Shaffer (1981), and accelerometers on the hammers and the keys, as used by Askenfelt and Jansson (1990). Mathews (1975) observed that tone overlap was required in order to produce a legato effect in tones generated by electroacoustic means. His observation was later corroborated by Purbrick (2000) who pointed out the difficulty in producing expressive performances with computer-controlled synthesizers, and proposed an automatic generation of legato articulation in a guitar sound generated with a physical model-based synthesis. In investigations of articulation in both digital and acoustic piano playing, Repp examined both perception and production of legato and staccato articulation. He found that an acoustic overlap was required to produce a legato, while a micropause was needed to produce a staccato (Repp 1995, 1997, 1998). Palmer (1989) reported that in legato articulation the inter-onset interval (101) between two overlapping notes is a major factor for the amount of overlap. Gabrielsson and Juslin pointed out how articulation, with its variations, is one of the most important and effective cues in communication and perception of motional-emotional character in music performance (Bengtsson and Gabrielsson 1983; Gabrielsson 1994, 1995; Gabrielsson and Juslin 1996). In the work presented in this paper a further step has been taken toward understanding the practice of articulation. It is a preliminary model for the rendering of articulation strategies in automatic piano performance. 2 Material and Method Two performance databases were used in this study. One consisted of performances by five diploma students, at the Venice Music Conservatory, who played the Andante movement of W A Mozart's Piano Sonata in G major, KV 545. They were asked to play the piece in nine different performance styles on a Disklavier connected to a PC. The styles were given in terms of adjectives (glittering, dark, heavy, light, hard, soft, passionate, flat, and natural, the latter meaning the style preferred by the pianist). The other database consisted of recordings of thirteen Mozart's piano sonatas played by a professional pianist on a B6sendorfer SE290 computer-monitored concert grand piano. Gerhard Widmer used the same database in research presented in these proceedings (Widmer 2001). The data available in the two databases allowed analysis of articulation based on the movement of the piano keys and not on the acoustic realization. Apart from the 101, two parameters have been used here for describing the articulation in piano performance. The first is the key overlap time (KOT), i.e., the time during which the keys corresponding to two successive notes are depressed simultaneously. The second parameter is the key detached time (KDT), defined as the time during with neither of two keys corresponding to two successive notes is depressed, such that there is a micropause between the tones (Figure 1).

Page  00000002 (a) (b) 3.1 Score legato articulation rule DM Lisp function name: Score-legato-art. Description: this rule produces an overlap of tones, or legato. The pseudocode of the rule is presented below. Affected sound parameter: key overlap time, KOT [ms]. Usage and limitations: the rule can be used to control the quantity of legato articulation. It is applied to notes that are marked legato in the score, as suggested by the name of the rule. Groups of legato notes are marked in the score file with the Lisp commands (LEGATO- START T) and (LEGATO-END T). (See the DM documentation for information about the score format and the Lisp commands available). Pseudocode for the Score Legato Articulation rule: if I <K<= 5 then KOT <- ((0.5-10-6-K - 0.1110-3)101I+ 0.01105-K + + 0.16063)-101 else if 0<K<= then KOT -- ((-4.3 10-6-K - 6.6-10-6).IO + + 58.533-10-3K K+ 113.15-10-3)I-0I where K is a weighting parameter determining the magnitude of KOT. The K values can be associated with the different playing styles corresponding to the adjectives used for the experiment described in a recent work by Bresin and Battel (2000): Figure 1. (a) Definition of inter-onset interval (IOIn), duration (DRn) and key overlap time (KOTn) for TONEn followed by an overlapping TONEn+1. (b) Definition of inter-onset interval (IOIn), duration (DRn) and key detached time (KDTn) for TONEn followed by a nonoverlapping TONEn+1. 3 Articulation Rules In previous research by Bresin and Battel (2000) and Bresin and Widmer (2000), measurements of the performances stored in the above-mentioned databases were statistically analyzed. From these analyses the following main results emerged; (1) KOT in legato notes is dependent from 101 of the first of two overlapping notes, (2) KDT in staccato notes is independent from 101, and (3) KDT in repeated notes is generally dependent from 101. These results led to the definition of a new set of rules for automatic articulation in expressive piano music. The new rules are four. They are included in Director Musices (DM), a rule system for music performance written in Common Lisp language (Friberg, et al. 2000). All the articulation rules contribute with different effects according to expressive indications (first database), tempo indications (second database; adagio, andante, allegro, presto, and menuetto), and articulation marks written in the score. The <Score legato articulation> rule is applied to notes marked legato. The <Score staccato articulation> rule is applied to notes marked staccato. The <Articulation of repetition> rule inserts a micropause between two consecutive tones with the same pitch. The <Duration contrast articulation> rule can be used to control the type of articulation, ranging from staccato to legato, of notes that are marked neither legato nor staccato. In the next paragraphs a description of these rules is provided. It follows the format used in the DM documentation (for more information see the list of links at paragraph 5). K=5 K= 1 K= 0.1 => passionate legato => natural legato => flat legato It means that by varying the emphasis parameter K from 0.1 to 5 it will be possible to sweep through a variety of legato styles. 3.2 Score staccato articulation rule DMLisp function name: Score-staccato-art. Description: this rule introduces a micropause after a staccato tone. The pseudocode of the rule is presented below. Affected sound parameter: key detached time, KDT [ms]. Usage and limitations. the rule can be used to control the quantity of staccato articulation. It is applied to notes marked staccato in the score, as suggested by the name of the rule. Staccato is marked in the score with the Lisp command (STACCATO T). An extra parameter, Tempoindication, can be used to achieve different quantities of staccato for different tempo indications. The DM command line for the Score Staccato Articulation rule is therefore: Score-staccato-art <K>:Tempo-indication <tempo> where tempo is the user input value for the Tempoindication variable.

Page  00000003 Pseudocode for the Score Staccato Articulation rule: if 1<K<= 5 then KDT -- (0.0216-K + 0.643) -101 else if 0<K<=1 KDT -- (0.458-K + 0.207) -IOI KDT -- pitch-contour - context - Tempo-indication - KDT where K is a weighting parameter determining the magnitude of KDT, pitch-contour, context and Tempoindication are three variables realizing the effects due to pitch contour, staccato context and tempo indication as described by Bresin and Widmer (2000). In particular they found that (1) KDT is larger for repeated notes and smaller for ascending pitch contours, (2) KDT is larger for isolated staccato notes and smaller for notes in a sequence of staccato notes, (3) KDT is smaller for slower tempi. The K values are associated with the different playing styles given below, corresponding to the adjectives used in a previous experiment on expressive intentions described by Bresin and Battel (2000): K K K K K K 5 3 1 0.6 0.5 0.1 default staccatissimo (KDT = 0.75-IOI) light staccato natural staccato (KDT = 0.5101O) default staccato heavy staccato default mezzostaccato (KDT = 0.25101) Pseudocode for the Repetition ofArticulation rule: if Expr = constant-kdt then KDT-- 20-K; else if Expr = varying-kdt if K>I then KDT-- ((-46-10-6.I0I- 23.67-10-3) -K- 878-10-6.10I+ 0.98164) -101 else if K<=l then KDT-- ((- 532-10-6.101+ 0.3592) -K248-10-6.10 + 0.3578) -IOI where K is a weighting parameter determining the magnitude of the rule effect. The Expr and K values are associated with the different playing styles given below corresponding to the adjectives used for the experiment reported by Bresin and Battel (2000), shortly described above: if Expr = constant-kdt then: K= 1 ==> natural K=0.7 ==> heavy if Expr = varying-kdt then: K=5 => dark K=4 ==> soft K= 2 ==> passionate K 1 ==> glittering K= 0.5 ==> flat and light K=0.1 ==> hard 3.4 Duration contrast articulation rule The first version of this rule was presented in Bresin and Friberg (1998). The current, slightly modified version is described here. DM Lisp function name: Duration-contrast-art. Description: the rule inserts a micropause between two consecutive tones if the first note is a short one, i.e., if it has duration between 30 and 600 milliseconds (see Table 1). Affected sound parameter: offset-to-onset duration, DRO [ms]. Usage and limitations: this rule can be used for the purpose of articulation, as suggested by its name. It can also be inverted, in the sense that it produces overlap of tones, or legato. Thus, the rule can be used to control the type of articulation, ranging from staccato to legato. It applies to notes which are marked neither legato nor staccato in the Table 1. Relation between tone duration (DR, in ms) and offset-to-onset duration (DRO, in ms) according to the Duration Contrast Articulation rule. Default value for both variables pitch-contour and context is 1. Their values can be automatically modified by DM, according to the findings by Bresin and Widmer (2000), shortly described above. The Tempo-indication values are associated with the different tempi given below (Bresin and Widmer 2000): Tempo-indication Tempo-indication Tempo-indication 1.3 1.15 1 => Presto and Menuetto => Allegro => Adagio and Andante 3.3 Articulation of repetition rule DM Lisp function name: Repetition-art. Description: the rule inserts a micropause between two consecutive tones with same pitch. The pseudocode of the rule is presented below. Affected sound parameter: key detached time, KDT [ms]. Usage and limitations: the rule inserts a micropause between two consecutive tones with the same pitch. An expressive parameter Expr can be used to achieve two different kinds of articulation, one with constant KDT, the other with KDT dependent on IOI. The DM command line for the Articulation of Repetition rule is therefore: Repetition-art <K>:Expr <kdt-type> where kdt-type is the user input value for the Expr variable. DR < 30 200 400 > 600 i DRO 0 -16.5 -10.5 0

Page  00000004 score, as such notes are processed by the Score Legato Articulation and the Score Staccato Articulation rules. The rule is not applied to the first tone of tone repetitions. 4 Conclusions The effect of these rules has not yet been tested with a listening test. Nevertheless it is equal to or larger than the just noticeable quantity necessary for perceiving legato or staccato tones according to findings by Woody (1997). The analysis-by-synthesis method confirmed the importance of the articulation rules to the improvement of the quality in automatically generated performances. The articulation rules were successfully used in the production of emotionally expressive performances of different scores, which where classified as happy or sad in accordance with hypotheses by Juslin, Friberg and Bresin (in press). The articulation rules are being applied in the design of control models for sound synthesis (Bresin, R., Friberg, and Dahl submitted), the main idea is to provide a more natural and realistic control to synthesis algorithms (Dannenberg and Derenyi 1998). The rules will be further developed in the near future. In order to achieve a more complete model of articulation, the acoustic quantity corresponding to key overlap and key detached times in legato and staccato respectively is under investigation. 5 Links The art of music performance: http://www. speech.kth. se/music/performance Articulation rules and sound examples: http://www. speech.kth. se/music/performance/articulation The Director Musices program: http://www.speech.kth.se/music/performance/download/ 6 Acknowledgments This work was supported by the European Union (SOb - The Sounding Object project - no. IST-2000-25287; http://www.soundobject.org), and by the Bank of Sweden Tercentenary Foundation (FEEL-ME - Feedback Learning in Musical Expression - Dnr 2000 - 5193:01; http://www.psyk.uu.se/forskning/projekt.html#pj). This research was also funded in part by the START programme Y99-INF, financed by the Austrian Federal Ministry for Education, Science, and Culture (http://www.ai.univie.ac.at/ /oefai/ml/music/musicproj ect.html). References Askenfelt, A. and E. Jansson. 1990. "From touch to string vibrations. I: Timing in the grand piano action." Journal of Acoustical Society of America, 88(1), 52-63 Bengtsson, I. and A. Gabrielsson. 1983. "Analysis and synthesis of musical rhythm." In J. Sundberg (Ed.), Studies of music performance, Stockholm: Publications issued by the Royal Swedish Academy of Music, 39:27-59 Bresin, R., A. Friberg, and S. Dahl. Submitted. "Toward a new model for sound control" Bresin, R., and G.U. Battel. 2000. "Articulation strategies in expressive piano performance. Analysis of legato, staccato, and repeated notes in performances of the Andante movement of Mozart's sonata in G major (K 545)." Journal ofNew Music Research, 29 (2000), No 3, 211-224 Bresin, R. and A. Friberg. 2000. "Emotional Coloring of Computer-Controlled Music Performances." Computer Music Journal, 24:4, 44-63 Bresin, R., and G. Widmer. 2000. "Production of staccato articulation in Mozart sonatas played on a grand piano. Preliminary results." Speech Music and Hearing Quarterly Progress and Status Report, Stockholm: KTH, 2000(4):1-6 Dannenberg, R., and I. Derenyi. 1998. "Combining Instrument and Performance Models for high-quality Music Synthesis." Journal ofNew Music Research, 27(3):211-238 Friberg, A., V. Colombo, L. Fryden, and J. Sundberg. 2000. "Generating Musical Performances with Director Musices." Computer Music Journal, 24(3):23-29 Gabrielsson, A. 1994. "Intention and emotional expression in music performance." In A. Friberg, J. Iwarsson, E. Jansson & J. Sundberg (Eds.) Proceedings of the Stockholm Music Acoustics Conference 1993, Stockholm: Royal Swedish Academy of Music, 108-111 Gabrielsson, A. 1995. "Expressive intention and performance." In R. Steinberg (Ed.) Music and the Mind Machine: the Psychophysiology and the Psychopathology of the Sense of Music. Berlin, Heidelberg, New York: Springer Verlag, 35-47 Gabrielsson, A., and P.N. Juslin. 1996. "Emotional expression in music performance: between the performer's intention and the listener's experience." Psychology of Music, 24:68-91 Juslin, P. N., A. Friberg, and R. Bresin. In press. "Toward a computational model of expression in performance: The GERM model." Musicae Scientiae. Mathews, M.V. 1975. "How to make a slur." Journal of Acoustical Society ofAmerica, Vol. 58 (Sl), S132 Palmer, C. 1989. "Mapping musical thought to musical performance." Journal of Experimental Psychology: Human Perception and Performance, 15:331-346 Purbrick, J. 2000. "Automatic Synthesizer Articulation." Computer Music Journal, 24(1):20-31 Repp, B. 1995. "Acoustics, perception, and production of legato articulation on a digital piano." Journal of Acoustical Society ofAmerica, 97(6):3862-3874 Repp, B. 1997. "Acoustics, perception, and production of legato articulation on a computer-controlled grand piano." Journal of Acoustical Society ofAmerica, 102(3):1878-1890 Repp, B. 1998. "Perception and Production of Staccato articulation on the Piano." Unpublished manuscript, Haskins Laboratories, http://www.haskins.yale. edu/haskins/STAFF/repp.html Shaffer, L. 1981. "Performances of Chopin, Bach and Bartok: studies in motor programming." Cognitive Psychology 13:326 -276 Widmer, G. 2001. "Inductive learning of general and robust local expression principles." In Proceedings of the International Computer Music Conference 2001, ICMA: San Francisco Woody, R.H. 1997. "Perceptibility of changes in piano tone articulation." Psychomusicology, 16:102-109