Page  167 ï~~An Automatic Jazz Accompaniment System Reacting to Solo Isao Hidaka Masataka Goto Yoichi Muraoka School of Science and Engineering, Waseda University 3-4-1Ohkubo Shinjuku-ku, Tokyo 169, JAPAN. +81-3-3209-5198/{hidaka,goto,muraoka} @muraoka.info.waseda.ac.jp ABSTRACT: This paper presents an automatic accompaniment system for jazz that understands solist's intention of improvising and outputs expressive accompaniment of a bass guitar and drums reacting to solo. Previous systems extracted just one parameter as solist's intention and output accompaniment by switching registered patterns. Our system can understand solist's intention in more detail by extracting five parameters as his intention fiom solo. In addition, to make accompaniment rich in variation, this system directly alters accompaniment data such as loudness, pitch and the number of notes, according to those parameters. In our experiment, it outputs appropriate accompaniment following the solist's intention. 1 Introduction This research aims to simulate human music listening process in a jazz ensemble, and cooperation among players. Although there are various kinds of jazz, we consider only four-beat jazz standards with a constant tempo. We deal with trio play that consists of a human pianist (or a guitarist), a bassist and a drummer by a computer. In this study we assume that the most significant player is a pianist, and the other two are accompanists - i.e., the pianist improvises a solo with a certain intention, which plays very important role in this ensemble. If the accompanists do not understand it, the ensemble sounds poor totally. It is therefore desirable that the accompanists play following the solist's intention. A previous system called JASPER [Wake et al., 1994] extracted just one parameter called tension parameter as solist's intention and output the accompaniment by switching pre-registered patterns according to this parameter. It was not enough to express his intention with only this parameter, however. In another system, Tsutomu Kanamori [Kanamori et al., 1993] implemented a listening model in a jazz session. They proposed detecting four kinds of sensuous information from the performance as a message that represented a psychological state of a player. In other related work, Robert Rowe implemented an interactive music system called Cypher [Rowe, 1992]. This system classified MIDI input into six features such as loudness, duration, and density, to understand a human's play. However, it didn't extract player's intention. Our system can understand solist's intention in more detail by extracting multiple parameters as his intention from the various aspects of the solo. In addition, to make the accompaniment rich in variation, this system alters the accompaniment data such as loudness, pitch, and the number of notes, according to these parameters. This system takes MIDI data as input, plays based on stored data such as keynote, chord progression, theme and plain accompaniment data, and outputs MIDI data as the accompaniment. Our system extracts five parameters called intention parameters as solist's intention from solo performance. In our current implementation, the intention parameters are excitement, tension, emphasis on chord, chord substitution, and theme reprise. Since it is hard to extract these parameters from individual notes directly, the system first detects eight musical primitives from the notes. These are louder note, chord tone, higher note, and so on. Each intention parameter is induced by summing up the particular musical primitives. Our system then alters the stored accompaniment data according to the intention parameters. The intention parameters control two modules that directly alter the accompaniment data. One module called MIDI filter adjusts loudness and pitch. The other called MIDI phase shifter adds notes by shifting timing of notes. This system has been implemented on a personal computer and works in real time. It was tested by a pianist and a guitarist. Experimental results show that it was able to extract intention parameters from individual notes of solo, and output appropriate accompaniment of a bass guitar and drums that followed the solist's intention of improvising. 2 Features of our system 2.1 Multiple solist's intention Extracting various solist's intention enables the system to understand the solo and to react to it appropriately. We therefore extract solist's intentions that are important for jazz play as five intention parameters from various aspects of solo (Figure 1). The five intention parameters are as follow: I CMC P ROCE E D I N G S 199516 167

Page  168 ï~~" excitement The parameter excitement expresses a solist's intention to make the ensemble inspiring. This intention is significant in jazz as well as in other music genres. This parameter is similar to JASPER's parameter. The accompanists excite their play reacting to this intention. " tension The parameter tension expresses a solist's intention to make the ensemble a little different from a standard by playing some notes such as tension notes and higher notes. Since tension notes are non-chord notes, they cause a variety of chord feelings. For example, when our system detects many tension notes, this intention is extracted. " emphasis on chord The parameter emphasis on chord expresses a solist's intention to emphasize a standard's chord progression by playing a phrase that includes some chord notes specified in a song. For example, our system extracts this parameter when the solist plays a phrase such as arpeggios of the chord notes. " chord substitution The parameter chord substitution expresses a solist's intention to emphasize standard's chord progression with slightly different from a song by playing a phrase that includes some substituted chord notes. Because a substituted chord has the same function as the original chord has, the solist can play a phrase with some substituted chord notes with keeping a feeling of the original chord. " theme reprise The parameter theme reprise expresses a solist's intention to plays a standard's theme or a faking theme that can be recognized as the standard. When the system detects this intention, it outputs plain accompaniment data without any alteration. 2.2 Dynamic alteration of accompaniment To make a total ensemble better, it is necessary for the accompanists to play reacting to the solist. We dynamically alters accompaniment data such as loudness, pitch and the number of notes, according to the intention parameters mentioned above. Even if the solist suddenly changes his play, the system can make accompaniment more appropriate to solo than outputting by switching pre-registered patterns. Figure 1 shows five intention parameters, musical primitives that induce intention parameters, and alteration of accompaniment reacting to intention parameters. In the next section, we describe how the system works in more detail. 3 System description Musical Primitives Intention Paranters Alteration of accompaniment louder note louder note higher note excitement cymbals S many notes many notes tension note higher note higher note noteteon [ tension note tension note chord note chord note emphasis on chord tension note tension note note in a scale.... d chord substtutoon.... other chord note other chord note themenote theme reprie plain accompaniment Figure 1: Solist's intention Figure 2 shows an overview of our system. The system consists of three main stages, Detection of musical primitives, Extraction of intention parameters, Alteration of accompaniment. For the preparation of a real-time performance, chord and key are analized in the Analysis of chord and key stage, in advance. In the Detection of musical primitives, eight musical primitives are obtained from the solo's individual notes using the results of analysis of chord progression and key. In the Extraction of intention parameters, five intention parameters are induced by summing up the corresponding musical primitives. In the Alteration of accompaniment, accompaniment data are directly altered according to the intention parameters by using two modification modules. 3.1 Analysis of chord and key The system analyzes the chord and key on the basis of music theory [Taniguchi, 1991] in order to understand chord function, tension notes in the stored chord progression, available note scale and so on. The system utilizes these results to detect the musical primitives from solo performance and to alter accompaniment data. 168 IC M C PROCEEDINGS 1995

Page  169 ï~~accompaniment of bass and drums Figure 2: Overview of our system 3.2 Detection of musical primitives For the preparation of extracting the intention parameters, the system detects eight musical primitives from every note to understand some of musical features of the notes. The musical primitives tension note, chord note, scale note, and other chord note are detected by using the results of analysis of the chord and key. Another primitive theme note is detected by matching the input note with the theme melody. This stage does nothing for the other primitives louder note, higher note, and many notes, since they are simple primitives, and can be obtained directly. 3.3 Extraction of intention parameters mIntention parameters m musil excite tension emphasis chord sub- theme In this stage, five intention parameters are ex- primitives on chord stitution reprise louder note 100 ---..... tracted from musical primitives. Each intention higher note 80 70-........ parameter is induced by summing up the par- many notes 5 -- ticular musical primitives shown in Figure 1 tension note 3_ - _ 0. To follow a solist correctly when he suddenly note in a scale --- --- - 0 -- changes his intention, this system extracts these other chord note --- -. -- 0 --- parameters on every beat, and to extract a slight theme note -... _-- --,, but continuous intention, this system sums up Table 1: Threshold values for extracting intention parameters the musical primitives in a certain period. Our system weights some musical primitives according to their importance in the intention parameters under consideration. In summing up the primitives, our system ignores the primitives whose values are less than their threshold value. These threshold values are listed in Table 1. In this table, the value at the louder note is MIDI velocity, and the values at the higher note are MIDI note number. The mark * indicates that the parameter theme reprise is detected on the basis of how the input notes are matched with the stored theme without threshold. 3.4 Alteration of accompaniment In this stage, our system dynamically alters the accompaniment data according to values of intention parameters and the period for which the intention parameters extracted. To follow only the solist's apparent intentions, the system alters accompaniment data when an intention parameter's value is greater than it's threshold value. The intention parameters control two modules that alter accompaniment data directly. One module called MIDIfilter adjusts loudness and pitch. The other called MIDIphase shifter puts additional notes by shifting timing of the original notes. In this case, the intention parameters control how long and how many times IC M C P R OC E ED I N GS 199516 169

Page  170 ï~~it shifts data. For example, when the intention parameter excitement is extracted, the system makes the accompaniment note louder and higher with MIDI filter and it adds some notes with MIDI phase shifter. In the case of altering the bass part, the pitch of notes is calculated on the basic of the results if the analysis of the chord and key. In the case of altering the drums part, notes of other percussion instruments are added, or the part of a percussion instrument in the stored accompaniment is replaced by another instrument. 4 Experimental results excitement This system has been implemented on a personal com-.,1_... _.i puter (NEC PC-9821 Ap2) and works in real time. It o0 40 60 o - -100 64wc-......drums: added notes was tested by a pianist who had studied music in the. drum,: added cymbals university, and a guitarist who had experience of play- tension - -bass: added oes ing in an amateur jazz band. They played with a MID I,.i synthesizer (KORG 011W) or a guitar that can output 0 20 106oo 10,sÂ~co. MIDI streams. The accompaniment sounds are output - - - - -bass: higher notes by the synthesiser. We handled a standard called "take emphasis on chord -bass: tension notes the A train." We requested them to play the theme in the first half and then improvise. 0 20 40 60 8o 100 t ris.]nta Figure 3 shows how our system extracted five in- -_..- bass: chord otrd tention parameters, and how it altered accompaniment chord substitution data according to these parameters. Both players eval- 0 20(,, 40 60 80,, - 100 nt 0) players00 001z00 60 soc c. uated that the system output the accompaniment at an - - bass: scale notes amateur level, and that it could nearly extract their theme reprise intentions. The system sometimes extracted several in- o " o,0". AL* Li,, u 0 20 o40 60 90s 0 0loo sm(sc. tentions when the solist played with only one intention. -...-. piain accompaniment The reason for the mistakes is that a certain note caused Figure 3: Intention parameters and accompaniment the multiple musical primitives to be obtained. One of the solists pointed out that the system reacted too sensitive. This was chiefly because we fixed the threshold values that affected the extraction of intention parameters. We think that the system will react more appropriate to solo by dynamically modifying these values or by changing them so that they fit the solist. 5 Conclusion In this paper, we proposed the way of extracting five parameters that express solist's intention and accompanying him appropriately by altering the accompaniment data according to them. To extract parameters as intention, the system summed up several musical primitives, and to alter the accompaniment data directly, we implemented two modules that handled raw MIDI data. Experimental results showed that our system was able to extract the intention parameters and output appropriate accompaniment reacting to them. We implemented a fixed accompanist because we statically defined all the threshold values concerning with extraction of intentions. We plan to improve the algorithm for modifying those values automatically during the performance. Future work will include the implementation of changing the accompaniment actively, and understanding solist's feelings. References [Wake et al., 1994] Sanae Wake, Hiroaki Katou, Naoki Saiwaki, Seiji Inokuchi: Cooperative Musical Partner System Using Tension Parameter: JASPER (Jam Session Partner) Trans. of Information of Processing Society of Japan, Vol. 35 No. 7 pp. 1469-1481, 1994 (in Japanese). [Kanamori et al., 1993] Tsutomu Kanamori, Haruhiro Katayose, Seiji Inokuchi, Hiroshi Hirai, Yasuhisa Niimi: Interpretation of Musicality in Jazz Improvisation using Multi-Agent Model, Proceedings ofIAKTA/LISTInternational Workshop on Knowledge Technology in the Arts, pp.107-114, 1993. [Kanamori et al., 1995] Tsutomu Kanamori, Haruhiro Katayose, Yasuhisa Niimi, Hiroshi Hirai, Seiji Inokuchi: A Study of Music Recognition in a Jazz Session System, Trans. of Information of Processing Society of Japan, Vol. 36 No. 1, pp. 139-152, 1995 (in Japanese). [Rowe, 1992] Robert Rowe: Machine Listening and Composing with Cypher, Computer Music Journal, Vol. 16, No. 1, pp. 43-63, 1992. [Taniguchi, 1991] Hiroshi Taniguchi: The Real Jazz Guitar Volume 1, Toky.o Ongaku Syoin, pp. 15, 28-34, 57-70 1991 (in Japanese). 170 0ICMC PROCEEDINGS 1995