Page  00000001 IMUTUS - AN EFFECTIVE PRACTICING ENVIRONMENT FOR MUSIC TUITION Spyros Raptis Aimilios Chalamandaris Alexandros Baxevanis Institute for Language and Speech Processing (ILSP) Speech Technology Department Artemidos 6 & Epidavrou, GR-151 25, Athens, GREECE spy@vilsp.gr Anders Askenfelt Erwin Schoonderwaldt Kjetil Falkenberg Hansen Royal Institute of Technology (KTH) Dept. of Speech, Music and Hearing Lindstedsvigen 24, SE-100 44 Stockholm, SWEDEN andersa@speech.kth.se Dominique Fober Stphane Letz Yann Orlarey GRAME Centre National de Creation Musicale 9 rue du Garet, 69202 Lyon, FRANCE fober(@grame.fr ABSTRACT This paper presents some major results from the IMUTUS project'. IMUTUS was an RTD project that aimed at the development of an open platform for training students on the recorder. The paper focuses on one of the most important and innovative parts of the IMUTUS system, the practicing environment. This environment integrates technological tools for the automatic analysis and evaluation of student performances along with enhanced interaction schemes to provide an effective approach to music learning. Testing and validation activities that have been carried out indicate that the IMUTUS approach is appreciated by both students and teacher, and that it clearly has a strong potential. 1. THE IMUTUS PROJECT The IMUTUS project aimed at the development of an open platform for training students on musical instruments. The recorder was chosen as the target instrument because it is a traditional instrument with no MIDI support, widely taught in European schools providing. IMUTUS is not intended to replace traditional forms of teaching. On the contrary, IMUTUS is expected to be most successful when used as a supplement to traditional music lessons with a teacher. It will be particularly effective at teaching the more repetitive, yet essential, aspects of learning to play a musical instrument. The system has been designed to address the needs of beginners (first level) between 9 and 14 years old. IMUTUS provides an attractive, engaging and easy-touse environment equipped with advanced technological tools to increase the effectiveness of music training, allowing the students to proceed at their own pace. Practicing sessions can be combined with theory lessons and fun games to create a complete learning setting. 1 This work has been partially supported by the European Community under the Information Society Technology (IST) RTD Programme, contract IST-2001-32270 - IMUTUS. The authors are solely responsible for the content of this paper. It does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of data appearing therein. Being an open learning platform, IMUTUS provides authoring tools for creating new educational units, allowing the content to be continuously augmented. Clearly, the main technological and pedagogical innovations of IMUTUS lie in the support it provides for student practicing, transforming it from a tedious task to a much more effective and enjoyable activity. The rest of the paper presents the approach adopted and the technology developed to support practicing in the IMUTUS environment. A distance learning dimension complements the IMUTUS core. This plays the role of a repository where the students can download new content, and a communication center where students can get in touch or upload their performances and ask for their teacher's feedback. Finally, a set of additional components are offered: a tuner to check the tuning of the recorder, a 3D viewer to demonstrate the fingering of a note, a movement authoring tool, a simple score editor used for ear training and similar activities, and an optical music recognition tool to scan music sheets and convert them to electronic format. IMUTUS has been designed to support different languages. Currently, English, Swedish, French and Italian are available, but any other language can be easily introduced. More information about the project and the consortium can be found on the IMUTUS website [1]. 2. A PRACTICING SESSION WITH IMUTUS It was recognized early in the IMUTUS project that the core functionality and value of the system mainly lies in support for the practicing sessions. This is the most relevant and innovative part of IMUTUS, both pedagogically and technologically. Various existing commercial systems claim to assist in instrument practicing, but in most of the cases their actual performance is far from satisfactory. The practicing sessions in IMUTUS exploit a rich set of technological components aiming to provide innovative and enhanced forms of feedback focusing on performance skills. The main components are the audio music recognizer, the score follower and matcher, the

Page  00000002 automatic performance evaluation module and the score viewer, described in separate sections below. An attractive, intuitive and simple-to-use graphical user interface has been designed to host the necessary interface elements and provide access to IMUTUS features and functionalities. IMUTUS provides a virtual tuner that can be used when warming up. This tool allows the system to for optimisation of pitch recognition. The practicing session starts with the student opening a music score from the available content. The student can use the tools provided by IMUTUS to see the score and listen to a respective MIDI or a reference performance recorded by the teacher. During playback, a running cursor indicates the current position in the score. For scores that span more than one page, IMUTUS "turns" the pages automatically when the end of the current page is approached, applying an efficient look-ahead strategy. When the student gets familiar with the piece, she can try to perform it. By pressing the appropriate button, IMUTUS starts recording, analyzing and following the student performance, displaying information about the current input level, the notes recognized and the identified position in the score. Following the student performance in real time, IMUTUS is able to provide automatic page turning while the student plays, when necessary. When the student has finished playing, she can request for evaluation of the performance. More accurate audio recognition and score matching algorithms are then invoked to provide detailed and complete analyses of the recorded audio, before proceeding to the actual evaluation. The evaluation results are presented through the user interface. They consist of an overall grading on a scale of 1 to 3 and the 3 most significant errors that have been detected. Figure 1 show a typical screen view of the IMUTUS practicing environment after a performance evaluation. In this case, the student has received grading 2 (two golden stars are displayed just above the score). After looking at the grading the student can press the error buttons (the light bulbs at the top right corner) one by one and read the error messages and hints. Here the student has pressed the second error button, and the system displayed the position of the error in the score (indicated by the arrows) and a descriptive message. A fingering error was made there by the student and the "Get help" button provides the feedback message: "Try to keep fingers as close to the flute as possible, even your left thumb." 2.1. Audio Recognition and Score Matching Dealing with a non-MIDI instrument such as the recorder, audio recognition and score following/matching are two indispensable tasks that I Figure 1. Screen view example of the IMUTUS practicing environment after a performance evaluation: The student has received grading 2 (two golden stars are displayed just above the score). After looking at the grading the student presses the error buttons (the light bulbs at the top right corner) one by on one and reads the error messages and hints. Here the student has pressed error button 2 and the position of the error in the score (indicated by the arrows) and a descriptive message are displayed. In this case a fingering error was made and the student presses the "Get help" button and reads the feedback message "Try to keep fingers as close to the flute as possible, even your left thumb."

Page  00000003 IMUTUS needs to perform in order to "understand" a student performance before being able to evaluate it. IMUTUS integrates two recognizers customized for the recorder, a real-time and an off-line, serving different purposes. During the student performance, the real-time audio recognizer and the score follower cooperate to identify and follow the current position in the score and provide the student with handy features such as automatic page turning/scrolling. The real-time recognizer is a highspeed, low-accuracy module that is only concerned with identifying notes. After the end of the student performance, the off-line audio recognizer is invoked. It is a high-accuracy recognition and analysis module responsible for segmenting the audio, accurately identifying notes and onsets, and extracting the features necessary for the actual evaluation of the performance. A final necessary step, is score matching. Its goal is to determine the correspondence between the recognized student performance and the reference music score. This problem is far from trivial, especially in the particular context of IMUTUS, since the score matcher will need to compensate for a number of "noise" sources, including: * Failures of the audio recognizer, e.g. errors of pitch recognition and spurious or missing note onsets and offsets, * Poor execution of a note by the student that could also distort the audio signal (e.g. over-blowing) or alter its spectral characteristics (e.g. leading to octave shifts), * Uncontrolled behavior of students during practicing, especially how they resume after making a mistake; they may try to repeat the same note, restart from the beginning of the phrase, or even restart from the beginning of the piece. The IMUTUS score matcher addresses all the above issues with a high degree of success. 2.2. Automatic Performance Evaluation The performance evaluation module (PEM) is activated on command of the student, directly after recording a performance. Performance evaluation plays a central role in the student-system interaction, making the practicing session more efficient. It does not only inform the students about their achievement, but also provides constructive feedback about their performance mistakes. The performance evaluation module looks for typical errors in the student performance. The error detection is based on a comparison between the performance and the reference score. Acoustical features are also taken into account. The errors are interpreted on a higher level, based on musical context and known difficulties in student performances. This enables the system to provide feedback which is closely related to performance skills. Except for errors in the performance, PEM can also provide positive feedback if the student played well (for example all the notes correct). In order to limit the amount of information to the student, a selection is made of the three most important performance errors. The selection criteria are based on a balance of negative and positive feedback, preference of high-priority errors (for example a missed note) and the order of importance of the performance skills, associated with the errors. All errors and positive remarks, which can be identified by PEM, are associated with a database of standard feedback messages. These are the messages, which are finally displayed in the score viewer. The student can choose if she wants to see a hint, or a more elaborate help message. A more detailed explanation of the performance evaluation process in IMUTUS can be found elsewhere in these proceedings [2, 3]. 2.3. Interaction with the Score Viewer The role of the score viewer is to facilitate the learning process by providing innovative and enhanced forms of feedback. To use a metaphor, the score viewer acts as a visual and sonic mirror that helps the student to become aware of her own performance. The interaction between the student and the score viewer can be divided in three main phases: In the explorative phase, the student will use the score viewer to read the score to be performed, to listen to the reference recording and take note of the difficulties as well as any indications on how to perform the score correctly. In the practicing phase, the student will use the score viewer to perform the piece by possibly using the metronome, the automatic cursor following and the turn-page facilities. In the feedback phase, the student will use the score viewer to become aware of her own performance through performance evaluation annotations, as well as the possibility to play it back, comparing it with the reference performance. In particular she will be able to directly click on a specific note or selected section in the score to listen to the corresponding audio recording. To provide the described features the score viewer is organized into a set of players in charge of the audio and MIDI rendering. The players are associated to a graphic module, providing the music score and related material, and supporting the user interactions. The music score is thus extended to become the main way to interact with the system and to provide feedback to the user. It supports pedagogical annotations, tips for a correct performance or for improvements, graphical signs to highlight a specific section, whether attached to a given exercise or

Page  00000004 dynamically computed by the system. Additional information is also related to the music on the score and is retrieved by interacting with the score. For example, a student may listen to a given note or ask for the fingering by clicking on the note. A more detailed description of the score viewer and its supported features can be found in [4]. 3. TESTING AND VALIDATION A set of validation activities have been planned and carried out for the IMUTUS prototype. The participants were students from 3 Swedish music schools, varying in age from 9 to 14 years. The students were distributed over an IMUTUS group (6 students) and a control group (6 students) with comparable musical experience. The IMUTUS group practiced 3-4 times a week at home using the IMUTUS prototype. The control group was instructed to record their practicing session on minidisk. The validation period lasted 3 weeks. At the end of each week the students of the IMUTUS group were asked to fill in a questionnaire. Also the three teachers were asked to fill in a questionnaire after the validation period. The validation addressed three different aspects of the prototype: affect and attitude, usability and musical improvement. The questionnaires showed that the IMUTUS group experienced working with the prototype as being fun throughout the validation period. The students also generally reported that the prototype was easy to use. An interesting shift of attitude was found for the teachers. While initially skeptical, they became interested in the IMUTUS concept during the validation period, when they noticed the effect on the students. There were also some indications that the IMUTUS group made better progress than the control group. For comparison, the recordings of the control group were also evaluated by the prototype in an equal manner as the recordings of the IMUTUS group. The results showed that the IMUTUS group displayed a larger and more persistent decrease of performance errors as a result of practicing than the control group. This effect was especially strong for fingering transitions. The differences could be explained by the fact that students using the prototype seemed to be more focused during practicing compared with the normal practicing situation. To summarize, the validation demonstrated that the prototype was fun to use for the students. The usability of the system was satisfactory. The prototype also seemed to have a positive effect on the musical improvement of the students. 4. REFERENCES [1] The IMUTUS website: http://www.exodus.gr/imutus [2] E. Schoonderwaldt, K. Falkenberg Hansen, and A. Askenfelt, "IMUTUS - An interactive system for learning to play a musical instrument", Proceedings of the International Conference of Interactive Computer Aided Learning (ICL), Ed. Auer, M. & Auer, U., Carinthia Tech Institute, Villach, Austria, 2004 [3] E. Schoonderwaldt, A. Askenfelt, K.F. Hansen, "Automatic evaluation of recorder performance in IMUTUS - design and implementation", Submitted to ICMC, 2005 [4] D. Fober, S. Letz, Y. Orlarey, A. Askenfeld, K. Falkenberg Hansen, E. Schoonderwaldt, IMUTUS - An Interactive Music Tuition System", Proceedings of the Sound and Music Computing Conference (SMC'04), IRCAM, Paris, pp.97-103, 2004