Page  1 ï~~I-MAESTRO: TECHNOLOGY-ENHANCED LEARNING FOR MUSIC Kia Ng,' Tillman Weyde, 2 and Paolo Nesi3 'ICSRiM - University of Leeds, School of Computing & School of Music, Leeds LS2 9JT, UK 2 City University London, School of Informatics, Department of Computing, Northampton Square, London N5 2TU, UK 3DISIT-DSI - University of Florence, Via S. Marta 3, 50139 Firenze, Italy info@ai-maestro.org www.i-maestro.or ABSTRACT In this paper, we present a project called i-Maestro (www.i-maestro.org) which develops interactive multimedia environments for technology enhanced music education. By exploiting new pedagogical paradigms with cooperative and interactive self-learning environments, gesture interfaces, and augmented instruments, this project builds on recent innovations resulting from the development of computer and information technologies and explores novel solutions for music training in both theory and performance. This paper discusses general context of the project along with the developments, together with an overview of the framework and highlights a number of different tools to support technology-enhanced music learning and teaching. 1. INTRODUCTION The i-Maestro project explores innovative solutions for technology-enhanced music education with a particular focus on bowed string instruments. Music performance is not simply to play the right note at the right time. Among the many challenging aspects of music education, we are particularly interested in linking music practice and theory training, looking at interactivity, expressivity and accessibility. Guided by an analysis of pedagogical needs, the project develops enabling technologies to support music performance and theory training, including tools based on augmented instruments, gesture analysis, audio analysis and processing, score following, symbolic music representation, cooperative support and exercise generation for tuition, self-learning, and collaborative work scenarios. i-Maestro offers a flexible, interactive multimedia framework and supporting tools which builds on recent innovations resulting from the development of computer and information technologies. 2. FRAMEWORK AND TOOLS With the continuous user requirements analysis, a set of use cases, test cases and pedagogical scenarios have been translated into specifications of the framework and tools. An overview of the framework is given in the diagram as presented in Figure 1. In this section, we highlight several i-Maestro tools that support different aspects of music learning and teaching. Figure 1. An overview of the i-Maestro framework. Sensors Posture and Visual Audio Accessible interface Gesture Rendering and Rendering and Interface interaction acquisition I-MAESTRO Music Training Exercise Processor Symbolic Practice Training Music Editing Assessment Training Processing Tools + Score- Models and Tools processing Tools Following I-MAESTRO Other players Cooperative Support Audio [DB access and tools for Music Training Processor: beat, pitch, etc. I-MAESTRO Client Tool i-Maestro I-MAESTRO School Production Tools To other MI-MAEST S ServerClient Cools t1 Teacher, Assessment Wxpertsok po Group of students I-MAESTRO P2P and Cooperative Work Support Educational individual Cot, students profiles, history Group of students 2.1. Symbolic Music Representation Music notation is fundamentals in music education. i-Maestro is promoting MPEG Symbolic Music Representation (SMR), an ISO standard for the representation of music notation with enhanced multimedia features [3, 4, 8, 9, 15]. Figure 2 shows an MPEG SMR player/decoder, which has been implemented within the IM1 MPEG-4 reference software.

Page  2 ï~~f liallffivils via f,i f Figure 2. An MPEG SMR player/decoder. 2.2. Cooperative Environment, Exercise Generator and the School Server Cooperative work is another key area of music education. It allows different components of the i-Maestro framework to be used across a network. Figure 3 shows an example student view of a collaborative ear training exercise. 1-maest4 SCHOOL SERVER mxw............................................................................................................................................................................................................................................................................................................................ 1..z.. #rne t ojc Tet choo Srvesaa Figure 5. i-Maestro School Server interface. Other tools include the Exercise Generator, which supports (semi-)automated creation of exercises (see Figure 4) and the School Server (see Figure 5) offers online access to stored lesson material for sharing learning material at home and in the classroom. 2.3. Music Training Supports The i-Maestro Sound and Gesture Lab includes advanced audio analysis, gesture- and score- following algorithms that provide feedback and accompaniment allowing new kinds of musical interaction (see Figure 6, left). The Gesture Follower can track a performed gesture in real time and compare it with pre-recorded gestures for a variety of pedagogical applications. The Augmented Violin allows bowing gestures to be tracked and studied (see Figure 6, right). Figure 3. Cooperative interface for ear training. Figure 6. T he Score F ollower listens to the player and provides automated "page turning" and accompaniment. Figure 4.Screenshots of the i-Maestro Exercise Generator. Figure 7. AMIR for 3D visualisation and sonification of a bowing exercise.

Page  3 ï~~4. ACKNOWLEDGMENTS The i-Maestro 3D Augmented Mirror (AMIR) [11, 13, 14, 17, 18] captures and visualises the performance in 3D (see Figure 7). It offers a number of different analyses to support the teaching and learning of bowing technique and body posture. The tool provides interactive multimodal feedback, online and offline with visualisation and sonification [19]. 2.4. Integration The combination of tools leads to new functionality, e.g. the automatic annotation of a score with bowing symbols in real time while a musician is playing, which is reached by combining, score follower, motion capture and SMR support. An application (called the i-Maestro Start) has been created to offer students and teachers a unique tool to start all the tools offered by i-Maestro. With the tools now available, validation is being carried out with teachers in music schools and conservatories. 3. CONCLUSION The project continued its work on pedagogical aspects, enabling technologies, software components, integrations and validation activities. An overarching pedagogical approach and model [16] for technology-enhanced teaching and learning has been developed. On this basis, a set of detailed pedagogical scenarios related to the use of the i-Maestro tools has been created. This paper presented a brief overview of the i-Maestro project. With the introduction, the paper presented the overall framework design and introduced several tools to support music learning and teaching including MPEG SMR for theory training, gesture analysis for performance training, with a particular focus on the 3D gesture and posture support using the 3D Augmented Mirror. The final results consist of a framework for technology-enhanced music training, that combines proven and novel pedagogical models with technological tools such as collaborative work support, symbolic music processing, audio processing, and gesture interfaces. Offering accessible tools for music performance and theory training will ensure wide participation. Prototype tools are now available and are expected to be incorporated in various new products and services, which will be made available to both the general public and educational establishments. These are being validated and refined and the project is inviting music teachers and students to take part in the validation phase. We are particularly interested in testing the system in real pedagogical situations to see how teachers and students interact with the technology. At the ICSRiM - University of Leeds (UK), open lab sessions are being organised for people to come and try out the i-Maestro 3D augmented mirror system with a 12-camera motion capture system. The research is supported in part by the European Commission under Contract IST-026883 I-MAESTRO. The authors would like to acknowledge the EC IST FP6 for the partial funding of the I-MAESTRO project (www.i-maestro.org), and to express gratitude to all I-MAESTRO project partners and participants, for their interests, contributions and collaborations. 5. REFERENCES [1] i-Maestro project website: www.i-maestro.org [2] i-Maestro project Deliverable DE4.5.1 on Accessibility aspects in Music Tuition, available online via http://www.imaestro.org/documenti/viewdocumenti.php?doci d=629 [3] P. Bellini, P. Nesi, G. Zoia. MPEG Symbolic Music Representation: A Solution for Multimedia Music Applications. Published on Interactive Multimedia Music Technologies, Copyright Â~ 2008 by IGI Global. ISBN 978-1-59904-150-6 (hardcover) - ISBN 978-1-59904-152-0 (ebook). [4] P. Bellini. XML Music Notation Modelling for Multimedia: MPEG-SMR. Published on Interactive Multimedia Music Technologies, Copyright Â~ 2008 by IGI Global. ISBN 978-1-59904-150-6 (hardcover) - ISBN 978-1-59904-152-0 (ebook). [5] Neil Mckenzie and David Crombie, Creating Accessible Interfaces for i-Maestro Learning Objects, in Proceedings of the Second International Conference on Automated Production of Cross Media Content for Multi-channel Distribution (AXMEDIS 2006), www.axmedis.org/axmedis2006, Volume for Workshops, Tutorials, Applications and Industrial, pp. 87-91, 13th - 15th December 2006, Leeds, UK, Firenze University Press (FUP), ISBN: 88-8453 -526-3, http://digital.casalini.it/8884535255 [6] Cont, A., Schwarz, D. (2006), Score Following at IRCAM, MIREX'06 (Music Information Retrival Evaluation eXchange), The Second Annual Music Information Retrieval Evaluation eXchange Abstract Collection, Edited by The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL), Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign http://www.musicir.org/evaluation/MIREX/2006_abstracts/MIREX20 06Abstracts.pdf, p. 94, October 2006, Victoria, Canada (http://ismir2006.ismir.net/) [7] F. Bevilacqua, N. Rasamimanana, E. Fl6ty, S. Lemouton, F. Baschet (2006) The augmented violin project: research, composition and performance report, 6th International Conference on New Interfaces for Musical Expression (NIME 06), Paris, 2006

Page  4 ï~~[8] Pierfrancesco Bellini, Paolo Nesi, Maurizio Campanai, Giorgio Zoia, FCD version of the symbolic music representation standard. FCD version of the symbolic music representation standard, MPEG2006/N8632, October 2006, Hangzhou, China [9] P. Bellini, F. Frosini, G. Liguori, N. Mitolo, and P. Nesi, MPEG-Symbolic Music Representation Editor and Viewer for Max/MSP, in Proceedings of the Second International Conference on Automated Production of Cross Media Content for Multichannel Distribution (AXMEDIS 2006), www.axmedis.org/axmedis2006, Volume for Workshops, Tutorials, Applications and Industrial, pp. 87-91, 13th - 15th December 2006, Leeds, UK, Firenze University Press (FUP), ISBN: 88-8453 -526-3, http://digital.casalini.it/8884535255 [10] Norbert Schnell, Frederic Bevilacqua, Diemo Schwarz, Nicolas Rasamimanana, and Fabrice Guedy, Technology and Paradigms to Support the Learning of Music Performance, in Proceedings of the Second International Conference on Automated Production of Cross Media Content for Multichannel Distribution (AXMEDIS 2006), www.axmedis.org/axmedis2006, Volume for Workshops, Tutorials, Applications and Industrial, pp. 87-91, 13th - 15th December 2006, Leeds, UK, Firenze University Press (FUP), ISBN: 88-8453 -526-3, http://digital.casalini.it/8884535255 [11] Ong, B., Khan, A., Ng, K., Nesi, P., Mitolo, N. (2006), Gesture-based Support for TechnologyEnhanced String Instrument Playing and Learning. International Computer Music Conference (ICMC) 6-11 November 2006, New Orleans, Louisiana, USA, www.icmc2006.org, ISBN: 0-9713192-4-3 [12] Bee Ong, Kia Ng, Nicola Mitolo, and Paolo Nesi, iMaestro: Interactive Multimedia Environments for Music Education, in Proceedings of the Second International Conference on Automated Production of Cross Media Content for Multi-channel Distribution (AXMEDIS 2006), www.axmedis.org/axmedis2006, Volume for Workshops, Tutorials, Applications and Industrial, pp. 87-91, 13th - 15th December 2006, Leeds, UK, Firenze University Press (FUP), ISBN: 88-8453 -526-3, http://digital.casalini.it/8884535255 [13] Kia Ng, Oliver Larkin, Thijs Koerselman, and Bee Ong, i-Maestro Gesture and Posture Support: 3D Motion Data Visualisation for Music Learning And Playing, in Proceedings of EVA 2007 London International Conference, Eds: Jonathan P. Bowen, Suzanne Keene, Lindsay MacDonald, London College of Communication, University of the Arts London, UK, 11-13 July 2007, pp20.1-20.8. [14] Kia Ng, Oliver Larkin, Thijs Koerselman, Bee Ong, Diemo Schwarz, Frederic Bevilacqua, The 3D Augmented Mirror: Motion Analysis for String Practice Training, p. 53-56, in Proceedings of the International Computer Music Conference, ICMC 2007 - Immersed Music, Volume II, pp. 53-56, 27 -31 August 2007, Copenhagen, Denmark, ISBN: 0 -9713192-5-1 [15] Kia Ng and Paolo Nesi (eds), Interactive Multimedia Music Technologies, ISBN: 978-1 -59904-150-6 (hardcover) 978-1-59904-152-0 (ebook), 394 pages, IGI Global, Information Science Reference, Library of Congress 2007023452, 2008. [16] Tillman Weyde, Kia Ng, Kerstin Neubarth, Oliver Larkin, Thijs Koerselman, and Bee Ong, A Systemic Approach to Music Performance Learning with Multimodal Technology Support, in Theo Bastiaens and Saul Carliner (eds.), Proceedings of E-Learn 2007, World Conference on E-Learning in Corporate, Government, Healthcare, & Higher Education, Qu6bec City, Qu6bec, Canada, Association for the Advancement of Computing in Education (AACE), October 15-19, 2007. [17] Thijs Koerselman, Oliver Larkin, and Kia Ng, The MAV Framework: Working with 3D Motion Data in Max MSP / Jitter, in Proceedings of the 3rd International Conference on Automated Production of Cross Media Content for Multi-channel Distribution (AXMEDIS 2007). Volume for Workshops, Tutorials, Applications and Industrial, i-Maestro 3rd Workshop, Barcelona, Spain, ISBN: 978-88-8453-677-8, 28-30 November 2007. [18] Kia Ng, Tillman Weyde, Oliver Larkin, Kerstin Neubarth, Thijs Koerselman, and Bee Ong, 3D Augmented Mirror: A Multimodal Interface for String Instrument Learning and Teaching with Gesture Support, in Proceedings of the 9th international conference on Multimodal interfaces, Nagoya, Japan, pp. 339-345, ISBN: 978-1-59593 -817-6, ACM, SIGCHI, DOI: http://doi.acm.org/10.1145/1322192.1322252, 2007 [19] Oliver Larkin, Thijs Koerselman, Bee Ong, and Kia Ng, Analysis and Sonification of Bowing Features for String Instrument Training, in Ng (ed), Proceedings of the 4th i-Maestro Workshop on Technology-Enhanced Music Education, co-located with the 8th International Conference on New Interfaces for Musical Expression (NIME 2008), Genova, Italy, ISBN: 978 0 85316 269 8, pp. 5-10, 4 June 2008 [20] Thijs Koerselman, Oliver Larkin, Bee Ong, Nicolas Leroy, Jean-Philippe Lambert, Diemo Schwarz, Fabrice Guedy, Norbert Schnell, Frederic Bevilacqua, and Kia Ng, SDIF Integration in iMaestro Gesture Tools, in Ng (ed), Proceedings of the 4th i-Maestro Workshop on TechnologyEnhanced Music Education, co-located with the 8th International Conference on New Interfaces for Musical Expression (NIME 2008), Genova, Italy, ISBN: 978 0 85316 269 8, pp. 15-20, 4 June 2008