Page  00000371 Virtual Conducting Practice Environment Guy E. Garnett, Fernando Malvar-Ruiz, Fred Stoltzfus (garnett, malvarru, fstoltzf} University of Illinois, Urbana-Champaign Abstract We have created an interactive system that allows conducting students of all levels to learn and practice with only a computer and a Buchla Lightningo. This environment targets both the student's stick technique and listening ability. The system uses real-time performance technology, including gesture tracking and recognition, score and tempo following, along with graphical and auditory feedback and cues. Keywords: Conducting, Computer-assisted instruction, Interactive Performance Systems, Music Education 1. Introduction During recent years, several systems have been developed that recognize and follow conducting gestures (Lee 92, Marrin 97, Morita 91). Such systems have been used mostly for performances that involve computer-generated music in combination with other instruments, or for furthering our understanding of what real conductors actually do, but rarely with pedagogical intention. We use a Buchla Lightning and a Macintosh computer to create a Virtual Conducting Practice (VCP) Environment. The Lightning is a MIDI controller (with builtin synthesizer) that consists of two batonlike transmitters, one for each hand, that send an infrared signal to a detector placed in front of the user. The Lightning transmits the position of the batons encoded as MIDI to the computer, the computer generates and sends back to the Lightning, or any attached MIDI device, MIDI data for aural feedback. 2. Conducting Feedback Our research has explored the nature of the feedback and the kind of userinterface environment that might give the best pedagogical results. Though in this phase of the project we aim at student users, the present work also serves as a test case for exploring feedback scores for non-student, that is, professional performers, to learn to control new, technologically-oriented, instruments and compositions. In this part of the project, however, we are not so much interested in how to control music synthesis by following conducting gestures-this indeed is fraught with all kinds of problems especially in the domain of control of tempo nuance and phrasingthough this is to some extent a required part of our system, rather we are trying to find useful ways to generate feedback to the conductor in order to help student conductors improve their understanding and their mastery of basic conducting technique. The main points of this research to be considered here are: 1) to determine a series of aural and visual representations (mixed-mode learning has been shown to be particularly effective) that will assist student conductors in understanding what they are doing right or wrong and how to improve or correct it; 2) to create a graduated, user-directed system that can focus on different aspects of conducting technique at different skill levels depending on the students needs, and can increase in difficulty as the student improves. This second point is indeed of crucial importance in creating a system that will be successful as a practice environment where the student will not have immediate access to an instructor and wishes to proceed at their own pace; ICMC Proceedings 1999 - 371 -

Page  00000372 2 *1 *3 2 --- Figure 1. showing four horizontal line plane. The Beat beats in 4/4 representing Window and the the beat that will be examined in a later section of this paper. First we turn, in the next section, to discussion of various aspects of representation we use in the system. 3. Representations Our system represents the gestures of the student conductor multiply, that is, we include not only feedback on various different aspects of their conducting gestures but also multiple representations of single aspects of their gestures. For example, tempo is represented by an audible beep at the point in time where the computer recognizes the ictus of the conducting gesture, but also it is represented graphically with sliders, in time versus tempo graphs, and with various. "instantaneous" (actually, averaged over beat periods) numerical values. This multiple representation is very helpful to the student; it allows a flexibility in their approach and use of the system. This flexibility allows the student to work in various "practice" modes, which can ultimately be contrasted with a "performance" mode. The former allows them to focus on particular parts of the overall exercise. At times, perhaps, for example, they might focus on maintaining an accurate and steady tempo, they will then make use of the extra precision of a visual representation, at other times they only need the background reinforcement provided by the audio cues. The "performance" mode, on the other hand, allows them to attempt to perform the exercise as a total piece and forces them to think of all of their technique simultaneously, much as they will have to in real performances. The primary visual representation the student will be faced with is the beat representation window. This window shows the reference beat plane (established as the initial vertical position of the baton when the student starts the exercise.) It then shows the ictus of the beats as large dots and shows the trajectory of the conducting gesture as curving lines (see Fig. 1). Simultaneously, the student can view various other windows displaying the same or different data in different ways. For example, a graphical slider object might show the student if the tempo is steady, increasing, or decreasing. Another object might show whether the student's beat plane is remaining stationary (as it should be) or is moving up or down or erratically. (The student in Fig. 1 has allowed their beat plane to creep upward from its initially established position, though the fourth beat, which has a tendency in most conductors to be slightly lower than the beat plane, has returned to just about on it.) Another window may show the consistency of the pattern height or other factors similarly displayed. 3.1 Aural Feedback Similarly, the gesture of the student generates aural feedback in multiple ways: first, as mentioned above, a beep can be sounded any time the student produces a downbeat. This sound can be contrasted with a different sound produced by the computer showing the student if the tempo is kept steady-a low note might sound on the "metronomic" beat, while a higher note might sound on the beat as derived from the student's pattern. The student can then practice "playing along with the metronome" until security of tempo is attained. The second kind of aural feedback is to present a - 372 - ICMC Proceedings 1999

Page  00000373 simple melody that follows the student's gesture in both tempo (derived from the time between downbeats) and loudness (derived in our initial studies strictly and rather too literally, from the vertical size of the beat pattern). The third basic type of feedback is generated from the overall shape of the gesture. For example, in the present implementation, the degree of legato or staccato is calculated based on the ratio of the maximum acceleration to the overall beat period: staccato beats have a very high acceleration (and corresponding deceleration) around the beat ictus for a given tempo, whereas legato beats tend to maintain a more uniform velocity throughout the beat curve. Aural feedback, in the form of shorter or longer beeps, is used to inform the student about the degree of legato implied by their beat pattern. 3.2 Music Representation In order to get beyond all but the most basic conducting situations, we have to be able to describe at least certain aspects of a "score" in a computer readable form. We have come up with a very simple representation that works quite well for the examples we anticipate in the early stages; remember, at this point we are not using the system for performance, but only for practice, therefore a limited score representation is perfectly acceptable. However, we do have to include a means for the computer to check whether the student is actually doing what they are supposed to be doing. We need to include tempo information, either qualitative (such as allegro, adagio) or quantitative (such as quarter equals 120). We also need to include metric information (4/4, 3/4, etc.), so we can check if the student is beating the correct pattern, and dynamic information (pp, f, etc.) so we can see if they are indicating that in their beats. We also include the possibility of a monophonic melody being played. For all this, it is simple enough to have a list with keywords and values, along with a system of defaults so that not everything has to be specified. At a later point we will revise this representation to be more inclusive and to allow us to specify realworld scores in some detail. That would also help in being able to work on the student's pitch-reading ability-the music can be prepared with errors to target the student's pitch recognition and discrimination. 4. Learning Curves Our method of utilizing multiple visual-aural representations is extremely useful for the student in determining what exactly is not being performed correctly and allows the student to focus their practicing on just those elements. Similarly, being able to start from absolute basics and work progressively at their own rate is very helpful to students. To this end, we have constructed several graduated exercises, such as you find in many conducting textbooks (McElheran). The easiest of these is simply to have the student "follow the beat" of the system's metronome, which is producible at various tempi. In order to do this correctly, the student must learn to make a basic beat pattern (in "one" to start with). This includes mainly just being able to move the wand up and down. Having aural feedback locating the computer's notion of the ictus is useful to assist beginners in understanding the motions they need to create in order to get the system to follow their beat. Tne system will show graphically and/or aurally both the metronome's and the student's beats. Once the student is able to correctly create the beat, they can progress gradually to learn different metrical patterns, using dynamics, changing tempo, etc. Some of these tools will also, we believe, prove useful for the more advanced conductor. The best example of this can be seen in situations of tempo modulation, erroneously called "metric modulation." In tempo modulation a performer is expected to change tempo based on a common subdivision between the different termpi. For -example, in quarter equals 60, to change smoothly and accurately to quarter equals 90, one thinks of triplets in 60 and then changes the beat to group every:wo ICMC Proceedings 1999 - 373 -

Page  00000374 of them together to get to 90. These, and especially more complicated variants, can indeed be difficult for even very experienced performers--our conducting trainer can help by at least showing the user how far and in what way they were off, something metronomes can not do. 5. Conclusions While we are just beginning a betatest phase with undergraduate beginners in conducting, we have tested the basic workings of the system on ourselves and random passersby. It is in no way a replacement for a good teacher, of course, nor for practice with live musicians for that matter, but we are confident it will be a useful tool for students. The advantages are potentially very beneficial. Any student conductor will have easy access to practicing with an "instrument" that responds to his/her motions much as an actual ensemble would. The increase in practice, with the appropriate feedback, will, especially for beginners, certainly be helpful. References Lee, M., G. Garnett, and D. Wessel. "An Adaptive Conductor Follower,"; Proc. of the 1992 ICMC. Marrin, T. And J. Paradiso. '"The Digital Baton: A Versatile Performance Instrument," Proc. 1997 ICMC, pp. 313-316. McElheran, Brock. Conducting Technique For Beginners And Professionals. Published: New York: Oxford University Press, 1989. Morita, H.,S. Hashimoto, and S. Ohteru. "A Computer Music System that Follows a Human Conductor." IEEE Computer Magazine, 24(7), July 1991, pp. 44-53. -374 - ICMC Proceedings 1999