Page  415 ï~~Experiments on Computerized Piano Accompaniment Shigeru IGARASHI Univ. of Tsukuba Takashi TSUJI Univ. of Tsukuba Tetsuya MIZUTANI Tsuyoshi HARAGUCHI Univ. of Tsukuba Univ. of Tsukuba Abstract. Our project has been studying computerized music accompaniment for ten years using the first grand piano controlled by minicomputers as well as the largest piano with MIDI-functions so far supplied by the leading piano manufacturer of Japan. In this paper, various subjects in our project are reported and discussed. 1 Introduction A conventional or acoustic piano played by a computer via MIDI data is still the most important instrument capable of automatically playing serious, or 'classic', music, especially when it comes to accompaniment, which involves many deep and subtle problems and, nevertheless, more usable in actual music scene than automated piano solos at least among musically trained people. Our project has been studying this theme for ten years using the first grand piano controlled by minicomputers as well as the largest piano with MIDI-functions so far supplied by the leading piano manufacturer of Japan. In section 2, development and experiments of a simple device using the 'click' of mouse handled by vocalists to control computerized piano accompaniment, and moreover, various styles of 'one-part' play, including 'one-finger' playing system will be reported. In section 3, we will report the analysis of typical 'phrase' expression, especially the displacement of speed and velocity in playing a phrase by good pianists. In section 4, the analysis of the change of speed of the practical music performance will be reported. In section 5, we will introduce a specification language for realtime systems, parallelism, logic and verification. Apparently, a computerized accompaniment system is a most subtle and severe realtime system so far encountered in computer science. Newer experiments employing MIDI-accordions are just to begin. We will introduce this topic in section 6. Conclusion will be found in section 7. 2 The Realtime Control System of the Automatic Performance When we perform a music, we change speed and velocity as the factor to express our emotion. The realtime control system of the automatic performance controls these two factors in realtime. We are developing the system using a mouse or two MIDI keyboards as control media, which is more convenient to control than Mathews' system [Mathews, 1989] using an electromagnetic drum. The system using the mouse displays a mouse cursor, computes speed magnification from an abscissa and velocity magnification from an ordinate, and controls speed and velocity (- dynamic) from the comn puted magnification. In order to change them suddenly, ten keys of a personal computer are assigned to the respective functions. As the other function, this system saves the coordinates of the mouse cursor corresponding to each event block of the original performance MIDI data, then it can replay the controlled performance. Our another system uses two MIDI keyboards, one setting the output to channel 1 and the other to channel 2. The information from the keyboard of channel 1 is used for the change of speed, and that of channel 2 for velocity. The function to control the balance of velocity between the right hand part and the left is assigned to a modulation wheel of the MIDI keyboard. This system can also save the controlled performance data in the form based on the MIDI standard. 3 Phrase Expression Using Characters of Performers We can use 'rendition' data to attach phrase expression to a performance. The rendition data consist of displacements of speed and velocity. Adding these data to the uniform mechanical performance obtained from a score, we obtain new data containing displacements of speed and velocity which are specified by the user of the system [Mizuta el al., 1992]. The rendition is considered as the difference between the score and the performance by a person. Thus, from the performance we can obtain the rendition which expresses the character of the player. First, let us plot an actual performance in a graph whose axis of abscissas is time and ordinates is the position on the score. The correspondence between time and the position on the score of each note is obtained as time of sound. Next, we express the displacement of time per beat as a graph whose abscissas is the position on the score and ordinate is time per beat. The second graph is no other than a derivative of the first graph. It must be noted that the axis of ordinate must be logarithmic scale from Weber-Fechner's law. This is important in the performance when it contains large displacement of time per beat. From the second graph we obtain a pattern of the rendition whose period is one over some integer. Let us assume that the period and whole length of the music is in the ratio of 1 to N. We sample the graph in 2" intervals for an integer ICMC Proceedings 1993 415 7P.04

Page  416 ï~~n and apply the fast Fourier transformation (FFT) to them. Then, the Fourier inverse transformation is applied to the component whose frequency is N times some integer, obtaining the pattern. We obtain the rendition data to apply a low-pass filter to this pattern. Assume that the data are expressed as M intervals. First, the pattern obtained as the above mentioned manner is concatenated with the reverse one and sampled in 2m intervals. Second, FFT is applied to it, the inverse transformation to the components whose frequencies are less than M, and the first half of the result is sampled in the M intervals (M +1 points). Then we obtain a sequence of the displacements of time per data which can be applied to the rendition data. This is a sum of sine curves each of whose frequency is an integer less than M/2, which approximates the original pattern. We analyzed 5 performances, played by the pianists Stanislav Bunin, Daniel Barenboim, Alfred Brendel, Vladimir Horowitz and Walter Gieseking, of the first 4 measures of the first movement of Mozart's piano sonata, K331. The input datum of each note is expressed as a pair of time value and the interval length to the next time of sound. From the analysis, we obtained some interesting results as follows: (1) The curve of Gieseking is similar to that of Brendel each of which has less fluctuation. Note that these two performers are recognized as the reputed Mozart players. (2) Except Gieseking's, the longest part of the time per beat, i.e, the slowest part, is the last of the 4th measure. Only Gieseking's curve has the slowest part in the first half of the 4th measure. In any case, the 4th measure tends to perform slow. This is considered as the manners of the performance of this music. (3) It is obvious that the expression of Bunin is larger than the others. Finally, we applied these results to the automatic music performance. Then we had an expressive performance but it is far from person's one. It is because expression is a combination of many elements but we use only two. 4 Analysis of Performance Patterns We analyze the change of speed of the practical music performances, especially those of the rhythm of waltz, using Taguti's method [Taguti, 1989]. Let f be the length of the actual performance at ith beat in measure j, which we normalize as fJ + f + fja -3, every j. Let fj be a three dimensional vector (f], f, ff). If all points f,'s gather into a small cluster on the plane G = {(z,y, z)I + y+ z = 3,xz,y,z > 01, then a point f representing the cluster is the unique agogic rhythm; if there are two such distinct clusters, then there are two distinct agogic rhythms; or if f3 's scatter over G, then there are no characteristic agogic rhythm. The following polar representation is useful to characterize possible agogic rhythms. Given f, define the quantity 0 = [A,B]: A = A(f) = If1 - f21 + f2- f3I + If3 - fll, arctan # if A >0, B- B(f)= 0 if A=0. Among such 0 = [A,B], put Cp = [1,(7r/8)p] (p = 0,..., 15) and fl = [0, 0]. We obtained the polar representations of agogic rhythms realized by 2 pianists (M, T) for Waltz No. 10 in B-minor Op. 69-2, Chopin. The dominating patterns are {7, 8, 9} in the pianist T. On the other hand, M's points gather around the center (IQ). Thus, we can recognize that M's performance is relatively faithful to the score. 5 Specification of Music Software In this section we will introduce a new verification system named tense arithmetic [Mizutani et al., 1993] using the concept of the analytic semantics [Igarashi, 1983]. Tense arithmetic is based on rational number theory, and deals with time as a rational value. It is based on the usual first-order predicate calculus with certain generalization of terms so as to include terms of the form A1;1...; A,, where A1,..., A are logical formulas. A generalized term A1;...; Am, called a sequential term, represents the first rational time when A,, holds after the first time of Am._. 1... after the first time of A1 after the observation time. We describe specifications and verify programs, especially in multi-processing and for realtime problems, using the arithmetic precisely and rigorously. We shall apply it to verifications of our music system. As an example, a program which accompanies a person is presented. We shall derive an inconsistency between the music theory and the first version of the program, and describe a correct specification in the music sense. First we define syntax and semantics of the tense arithmetic. The definition of the (conventional) terms are the same as the rational number theory. We consider a set of constants as that of program variables. Definition 1 If p is an n-ary predicate symbol and t,, t are terms, then p(ti,..., tn) is a formula. If A, Ai,...,Am,Bi,...,Bn are formulas andzx is a variable, then -'A, AI D A2,AI1VA2,AIAA2,VxA,HxzA and AI;...; Am -< B1;...; Bn are also formulas. 0l Expressions A1;.-.; Am -,, B1;...; Bn and A1; Â~.;Am -< B1;...; B, are abbreviations of (A1;...; Am Bi0;...;Bn) A (B1;...; BOn -_ Ai;...; Am) B1;...; B,), respectively. Symbols -<, -<,; and -. are called tense operators. 7P.04 416 ICMC Proceedings 1993

Page  417 ï~~Let E be a set of models of the original rational number theory. Let Q>0 be a set of all nonnegative rational numbers. Then a locus X is a mapping from E2 to Q>0. We now show the interpretation of formulas. Definition 2 For a formula A containing no tense operators, X, t = A if and only if x(t) k A. If A is a formula of the form Ax;...;Am B1;...;B,, then X, t = A if and only if for any y E (B1;...; Bn) -x, there exists x E (A1;...; Am) ~x,t such that x < y, where the set (A,;...; A,)X,t is {U I X, Uk 1 Ak,t < Uk <un = u,k = 1,...,m} and the set (Bi;...;B,) ~X,t is the similar one. 0 A formula A is said to be valid, written as [= A, if and only if X,t = A for any X and t. Let S be a set of time such that {t IX(t) # X(t + 0)}, where F[t+0] is an abbreviation of Ve > 0.36(0 < 6 <e A F[t + 6]) for a formula F. We suppose S is discrete, i.e., there exists e > 0 such that Ix - yj > e for any x, y E S. Namely, the set of time when the values of the program variables change is discrete. Furthermore, we consider a concept spur which is a generalization of a clock or a scheduler of each process of a parallel program. A spur a is defined by a = 1, where a is a constant treated as a program variable, and it satisfies the condition X(t) 0 X(t + 0) X, t 1 a, i.e., when a value of a program variable changes, the spur must be true, and vice versa. Now we consider a simple example for application of the arithmetic to an accompaniment system. Let us consider a simple score of triple time consisting of only one measure whose right hand part is (H, Q) and the left is (Q, Q, Q), where H is a half note and Q is a quarter. Let a and /3 be spurs of the right hand part and the left, respectively. Namely, each spur expresses the performance timing of each hand. Let C be the length of a quarter note of a local tempo computed by the system. For the sake of convenience, we use C in a sequential term like a; a; C, which means a sum of time of the second a and C. Note that it is easy to express this in a legal form. Let I denote (the timing of) the first bar. On the first version of the program, the specification for the left hand is as follows:;lm ^ i;&, I;/32,r;&;C, 1; 3 '. t;&2 and );3/4 I;3 (1) where & is an abbreviation of a; -,a, and &2 is of &; &. Let us consider the case where one plays in such a manner as I; &2.-<I; &; C holds. We can show the performance of the left hand is impossible since I;/.. j; &2.< J; &; C j;/ 2 and it contradicts j;/ 2 -< j;/ 3. Hence, a correct specification (from the view point of music theory) must be (1) if j; &;C -< j;&2 and l;i/ ""j;&, j;/ 2 J; &2 and I;I~ f,,; &3 otherwise. 6 Newer Experiments Employing MIDIAccordions Newer experiments employing MIDI-accordions are just to begin. Acoustic accordions obviously enjoy considerable capability of 'phrase' expression with 'bellows' compression, which information can now be supplied to computers as MIDI data. A full size MIDIaccordion specially supplied by a leading Italian manufacturer will be used as an instrument more or less close to a vocalist or a wind-instrumentalist, whose breath cannot be sensed or measured in a convenient manner at present. Additionally, a petit MIDI-accordion from the famous German company will be tested as a new 'conducting' device to control a whole music performance with its keyboard-button-bellows combination. 7 Conclusion We have reported and discussed various topics in our project, i.e., development of the realtime control system, analyses of phrase expression and pattern, and a specification language for realtime systems. All of these approaches, we think, are interesting and promising. References [Igarashi, 1983] Shigeru Igarashi. The v-Conversion and an Analytic Semantics. Inf. Proc. 83, R. E. A. Mason (ed.), Elsevier Science Publishers B. V. (North-Holland), IFIP, pp. 769-774, 1983. [Mathews, 1989] Max V. Mathews. The Conductor Program and Mechanical Baton. International Symposium on Music and Information Science, pp. 58-70, 1989. [Mizuta et al., 1992] Kiyoshi Mizuta, Kazuhiko Maruyama, Hiroki Komiyama, Shigeru Igarashi and Yasuhiro Ichinaka. Experiments with Computerized Piano. Programming Symposium, 33: pp. 103-112, 1992 (in Japanese). [Mizutani et al., 1993] Tetsuya Mizutani, Shigeru Igarashi, Hiroki Komiyama and Takashi Tsuji. Two examples of verification of parallel processes. Programming Symposium, 34: pp. 105-116, 1993 (in Japanese). [Taguti, 1989] Tomoyasu Taguti. Modeling and Analysis of Musical Performance on the Piano. International Symposium on Music and Information Science, pp. 71-78, 1989. ICMC Proceedings 1993 417 7P.04