Page  00000001 Psyche: University of Tsukuba, Computer Music Project Rumi Hiraga University of Tsukuba Shigeru Igarashi University of Tsukuba Abstract The computer music project Psyche started in 1983 at University of Tsukuba, AI lab. The project has been pursuing to let a computer render musically expressive performance especially on an acoustic instrument. Systems of Psyche are categorized into two. One consists of static programs to analyze and synthesize expressive performance. The other dynamic real-time programs to control a computer so as to accompany a human player. 1 Introduction The computer music project Psyche started in 1983 at University of Tsukuba, AI lab. Psyche's core research subject is to obtain musically expressive performance of classical music pieces by a computer. In order to pursue the objective, we have built and been building some infrastructures as well as individual computer music systems. Although the computer music has not a short history and some Japanese groups had been engrossed in the subject in 1983, the computer music as a research area has not been pervasive and recognizable in Japan in those days. We presented our prototype music description compiler language Europa (Extensible Universal Representation Of Phrasing and Articulation), which was used for representing information explicitly appeared on a score, at a symposium. The presentation inspired to formulate an informal computer music research group then now grown to an approved SIG (special interest group) of IPSJ (Information Processing Society of Japan). Professor Igarashi, an amateur tenor and an accordionist as well as a theorist in computer science [1], of the Institute of Electronics and Information Sciences of University of Tsukuba has been leading the project. About ten students (six undergraduate and four graduate) come to the project and are assigned systems to work on individual basis every year. They take over their systems from their seniors, on which they work for three years at the most (one year as an undergraduate and two in master program). Some of them make presentations on their systems at domestic conferences and demonstrate them at a Tsukuba musical festival every year. 2 Systems of Psyche In order to obtain musically expressive performance by a computer, many systems have been designed and built in Psyche in the following three directions. 1. To provide data for describing music score written in Europa beforehand to run any system. Systems with the score information are easy to extend when manipulation of performance becomes complicated. 2. To manipulate MIDI (strictly speaking, a special file format which is in accordance with MIDI), that has fewer performance parameters than data from other devices. So as to concentrate our research to the understanding and generation of musical expression with limited programming power (students), research subjects which are more immediate to synthesize and analyze music rendition are given priority. Consequently, some research issues of a few steps away from generating performance are not present objective, however important and interesting they may be. The research of automatic analysis of music structure, for example, has lower priority in Psyche, while users of Psyche systems analyze music structure for themselves to obtain performance. 3. To introduce music structure. This turned to be the most effective direction in solving various fundamental problems in music rendition. Several difficulties in automatic ensemble are solved elegantly by taking music structure into consideration.

Page  00000002 2.1 Equipment The acoustic grand piano with MIDI I/O and synthesizers are our oldest instruments for experiments. Other instruments with MIDI I/O are accordions, a silent drum system, a guitar, and a clarinet. The output of systems for synthesizing music rendition is played on the grand piano. The piano and other instruments are played by human players accompanied by Psyche's ensemble systems. Orchestrated data for piano concertos on the market are partly used for an accompaniment system also. Ensemble systems to control MIDI instruments have been built on DOS. Although informal, the inevitable time latency is pointed out by a developer of Windows 95, so that we have not yet decided to depend on Windows 95 to run the ensemble systems completely where the real time control is vital. The least time latency caused by an operating system is the biggest concern in selecting an OS on which accompaniment systems run. Static programs, such as synthesizing music rendition or visualizing musical performance, are written on Unix currently. 2.2 Current systems Systems of Psyche are categorized into two. One consists of static programs to analyze and synthesize expressive performance. The other dynamic real-time programs to control a computer so as to accompany a human player. The main systems are as follows. 2.2.1 Performance analysis and synthesis * Performance synthesis with rules. Musical rendition is obtained by unfolding performance of music structure (usually a motif) [8]. The performance expansion is derived by applying context-free performance rules based on music structure. Context-free rules are simple and describe relationships between occurrences of music structure and appearance of significant harmony progression. Currently, a portion of the actual performance by a professional pianist is used as a seed performance for the expansion. As for Mazurka Op. 7-2 by Chopin, providing one third of the whole performance as seed turned to be enough for synthesizing artistic music rendition. * Performance visualization for analysis and synthesis. There are several types of visualization systems in Psyche. All of them have proved that the characteristics are recognized when performance is presented visually based on music structure [2][6]. Figure 1 shows performance data of measures nine through twenty four of Chopin's Mazurka Op. 7-2 by two professional pianists. The visualization system gets MIDI data and the data on music structure, such as the information of motif or sentence. The performance starts from the leftmost fan shape, whose radius represents the length of a beat, and moves clockwise. A half circle represents a motif. Then the phase of the circle is inverted and the performance proceeds counterclockwise. We can observe several performance characteristics on this piece. Common ground in their performance. -Both of them play longer at the beginning and the end of a sentence which consists of four motifs. -The performance of the third motif resembles that of the first. We can recognize on the score that the two motifs are similar as music structure. Differences in their performance. -The length of the third beat in many of the motifs are played differently. A player plays it longer than adjoining beats, while the other plays it shorter. * Expression analysis on agogics and dynamics. In order to generate the seed performance used in the system of performance synthesis with rules, performance characteristics are analyzed and made into rules. MIDI data is used for the analysis. Example observations on several performance samples by professional pianists are as follows. - Although not recognized by listening, higher notes in Alberti bass played by the left hand are played louder and longer than other notes, contrary to the player's intention. - A note in the accompaniment part is played softer than other notes when some keys are pressed simultaneously. - Usually a note in the melody part is played first when some keys are pressed simultaneously. Human pianists are usually unconscious of these observations on performance. The quantification of performance rules which

Page  00000003 Figure 1: Comparisort of the performartce by two piartists are described qualitatively is ertabled by collectirtg more data. Attribute grammar is a cartdidate for a quarttificatiort method to get performartce data from qualitatively expressed performartce rules [3]. 2.2.2 Accompaniment systems * Ensemble for an acoustic grand piano. A humart player playirtg the primo part is accompartied automatically by the computer corttrolled secortdo part. Before the actual ertsemble, performartce by humart player is rehearsed to firtd out the basic tempo elasticity. Durirtg ertsemble, the system detects the tempo of humart performartce thert allticipates the player's tempo ill the rtext measure or motif [5]. The irtformatiort cortcerrtirtg music structure is giveR to the system as well as the irtformatiort of the score which ertables for the accompartimertt part to refer the way a humart piartist plays the corresportdirtg melody. The difficulties of the system is caused by the mechartical specificatiort of the grartd piarto that it must receive MIDI data SO0ms prior to the actual key movemertt. Cortsequerttly the calculatiort for the accompartimertt has to be dorte at least SO0ms earlier thart the actual key movemertt which is a severe restrictiort ill desigrtirtg the system. * An orchestra accompaniment to an acoustic grand piano. The orchestra accompartimertt system provides the automatic tempo corttrol facility, so that a piartist cart practice or ertjoy piarto cortcerto without botherirtg artybody else. * Ensemble with triggers. A computer corttrolled drum set with MIDI I/O accomparties art acoustic piarto played by a humart, or, cortversely, the acoustic piarto automatically follows the percussiort played by a humart. This system was the ortly exceptiort which didrt't use score irtformatiort provided irt advartce but ortly used triggers from the MIDI I/O device. Applyirtg "Art Der Schrtert, Blauert Dortau Op. 314" by Johartr Strauss II to the system has showrt the rtecessity of givirtg score irtformatiort, data of music structure, artd rehearsal data to the system for better performartce. 3 Future plans There are three subjects which are ort goirtg artd will corttirtue irt the future with marty system plarts irt Psyche. Three themes are as follows. 1. The design and implementation of music description languages Daphne and Leda. Music structure plays a very importartt role both irt performartce artalysis artd irt syrtthesis. Thus we have started the desigrt artd the implemerttatiort of a lartguage for represerttirtg music structure Daphrte (Declarative Artalysis of Pllrasirtg aNd Expressiort). Irt order to obtairt musically expressive performartce data, the irtformatiort ort music structure, relatiortships betweert occurrertces of music structure, etc. are rtecessary, which are writtert irt Daphrte. Performartce plarts based ort music structure is writtert irt Leda (Logical Expressiort of Dyrtamics artd Agogics). Usirtg the irtformatiort of music structure writtert irt Daphrte, systems look for performartce plarts at krtowledge base of corttext-free performartce method writtert irt Leda. 2. System integration by object-oriented approach. Besides takirtg advarttage of object-oriertted approach irt the desigrt artd the implemert

Page  00000004 tation of Psyche systems, object-oriented concept is adopted as the consistent design methodology. There are three reasons to use object-oriented approach. * To hold easiness of expansivity of systems and languages. It is often pointed out that music research has never reached to an end. A mechanism to append the attribute which is revealed later is necessary, since a particular attribute for music structure may be found to be desirable for expressive musical performance, for example. Inheritance mechanism is a method to expand or specify the later requirement to the system. * To integrate knowledge base into existing systems written in object-oriented programming language. Utilizing object-oriented approach for AI, Psyche systems can be seen in a single object-oriented manner as a whole. In order to represent knowledge base in an object-oriented way, we must take the latest research of the concurrent object-oriented issues into consideration. * To expand some systems as cooperative programming systems. For the purpose of evolving the current ensemble systems into cooperative programming systems, agents which run concurrently will be introduced. For this expansion, the concurrent objectoriented approach will be of good use. 3. The mathematical representation and formulation of musical performance especially for accompaniment systems. Accompaniment systems are regarded as real-time control systems whose states (positions, volume, etc.) are changing continuously but controlled by discrete actions, such as "note on" and "note off", by a program or a human performer. When the systems are specified formally, verification of correctness in program action and musical expression is mathematically and rigorously analyzed. For this purpose, we are formulating the systems in the verification formalism SOFA [7] based on the analytical semantics [4]. An example of system plans is an integrated musical performance editor. Psyche's systems are built mainly by expanding the existing ones. The editor has been designed and built based on performance visualization systems described above. References [1] Five Hundred Leaders of Influence, American Biographical Institute, 1997. [2] Hiraga, R., Igarashi, S., and Matsuura, Y.: Visualized Music Expression in an ObjectOriented Environment, Proc. of ICMC, pp. 483-486, 1996. [3] Hiraga, R., Igarashi, S., and Iyatomi, A.: Attribute Grammar for Music Processing System, 13th Conference Proceedings, JSSST, pp. 245-258, 1996 (in Japanese). [4] Igarashi, S.: The v-conversion and an analytic semantics, in Mason, R. E. A. (ed.), Inf. Proc. 83, pp. 769-774, 1983. [5] Igarashi, S., Tsuji, T., Mizutani, T., and Haraguchi, T.: Experiments on Computerized Piano Accompaniment, Proc. of ICMC, pp. 415-417, 1993. [6] Igarashi, S., Hiraga, R., Iyatomi, A., and Matsuura, Y.: Visualization of Music Interpretation Based on its Structure, Proc. of ICCMMS, pp. 21-26, 1996. [7] Igarashi, S., Mizutani, T., Shirogane, T., and Shio, M.: Formal analysis for continuous systems controlled by programs, Concurrency and Parallelism, Programming, Networking, and Security, Lecture notes in computer science, 1179, pp. 347-348, 1996. [8] Iyatomi, A. and Igarashi, S.: Making Computerized Piano Performance Artistic Reflecting Musical Structures, Proc. of the 9th Annual Conference of JSAI, pp. 621-622, 1995 (in Japanese).