Page  00000135 Cooperative Multimedia Environments for Technology-Enhanced Music Playing and Learning with 3D Posture and Gesture Supports Bee Ong', Ali Khan', Kia Ng', Pierfrancesco Bellini2, Nicola Mitolo2, and Paolo Nesi2 1 ICSRiM - University of Leeds, School of Computing & School of Music, Leeds LS2 9JT, UK 2 DISIT-DSI, Department of Systems and Informatics Distributed Systems and Internet Technology Lab, University of Florence, Via S. Marta 3, 50139 Firenze, Italy, Abstract This paper presents the I-MAESTRO project which is supported by the European Community under the Information Society Technologies (IST) Sixth Framework Programme to develop interactive multimedia environments for technology-enhanced music learning and teaching. It discusses the general architecture, using cooperative environments, and focuses on the gesture support aspects of the project. The project explores novel solutions for music training in both theory and performance with particular focus on string instruments, building on recent innovations resulting from the development of computer and information technologies, by exploiting new pedagogical paradigms with cooperative and interactive self-learning environments, gesture interfaces, and augmented instruments. Keywords gesture; technology-enhanced; string instrument; education; interactive multimedia; motion capture 1 Introduction to the I-MAESTRO Project The I-MAESTRO project (, is developing interactive multimedia environments for technology enhanced music education. The project aims to explore novel solutions for music training in both theory and performance, building on recent innovations resulting from the development of computer and information technologies, by exploiting new pedagogical paradigms with cooperative and interactive self-learning environments, gesture interfaces, and augmented instruments, with computer-assisted tuition in classrooms to offer technology-enhanced environments for ear- and practical-training, creativity-, analysis-, and theory-training, ensemble playing, composition, etc. I-MAESTRO is exploring Music learning practices developed over centuries by music educational institutions, and utilising technological advancements to enhance and amplify their qualities and propose possible new directions. Music educational institutions and other related actors can benefit from these new developments and solutions for increased efficiency, attractiveness, optimised learning outcomes and wider accessibilities, while reducing time and efforts. Music performance is not simply "playing the right note at the right time". The I-MAESTRO project is studying and exploring many aspects of music making in order to produce methods and tools for music education with innovative pedagogical paradigms, taking into account key factors such as expressivity, interactivity, gesture controllability and cooperative-work among participants. The main technical objectives of the projects include basic research and development on new solutions and enabling technologies to support traditional pedagogical paradigms for music training; novel pedagogical paradigms, such as cooperative-working, self-learning and classstudying, with particular focus on Symbolic Training (Mitolo et al. 2005) and Practice Training paradigms for string instruments exploring interactive, gesture-based, and creative tools; and a framework for technology-enhanced music educational models and tools to support the creation of flexible and personalisable e-leaming courses to improve accessibility to the musical knowledge (Crombie et al. 2005, Ng 2005a). 2 Framework A framework such as I-MAESTRO can assist Teachers in following the progress of each Student and in evaluating their achievements with the support of innovative methods. Using the tools in I-MAESTRO, Teachers will be able to create teaching material corresponding to the level of a Student or of a group of Students in a simple way, thus helping to personalise the tuition. The system also allows Teachers to monitor the Student's progress. In many teaching environments, physical contact and interaction with 135

Page  00000136 Students in classes are typically very limited in time and this can affect the quality of teaching. In this context, IMAESTRO provides for the creation of an almost continuous interaction between Students and Teachers, improving teaching continuity, and accelerating the rate of improvement. I-MAESTRO can also be very useful for supporting those Students who may need specific individual support. Students, Teachers etc. Sensors Posture and Visual Rendering Audio Rendering Accessible interface Gesture and interaction andcauisition Interface I-MAESTRO Music Training Exercise Processor ISymbolic Training Practice Training IIusic Editing + o Assessment I processing Tools II Processing Tools II Score Follower II Models and Tools DB access and tools Music Training beat, pitchjetc I-MAESTRO Client Tool I-MAESTRO To other L Production Tools I-M RO S\Group of studets,... Gro oEO Educational I-AESTRO Content profiles, Ps 2P and history and a results individual ooperative Srstudent Wuork Support (iv) I-MAESTRO P2P & Cooperative Work Support: for sharing content, supporting the cooperative work in classroom and among Students, stimulating collaboration and common experiences and setting up of groups of study, practice or theory training, ensemble training and playing, etc. 3 Musical Gesture: Posture and Gesture Support Musical gesture is perhaps a very much misused term with a wide range of interpretation. For this paper, by musical gesture, we simply mean the physical motion of playing an instrument, involving one or more parts of the instrument and/or the body of the player. Currently, posture and gesture support for conventional music training and learning is typically provided by using images and/or video. However, this is generally not effective due to the inherent limitations of 2D perspective views of the media. Playing an instrument is physical and requires careful coaching and training on the way a player positions himself/herself with the aim to provide the most economical output with least physical effort. In many ways, this can be studied with respect to sport sciences to enhance performance and to reduce self inflicted injuries. In this context, the I-MAESTRO framework is developing several related software modules for posture and gesture support. With these modules, a teacher can setup (either by recording or using some predefined gesture library of motion data) certain posture and gesture 3D graphics to support a particular training or specific exercise. It is recommended that this is done during a lesson when the teacher can access the student's physicality and capabilities. Figure 1. I-MAESTRO Architecture The I-MAESTRO General Architecture is presented in Figure 1. It includes: (i) I-MAESTRO School Server: An area for Lesson distribution, pedagogical material, history of Students, training material, tools for the Teachers, historical data collection and navigation, music school training management for the different scenarios. (ii) I-MAESTRO Client Tools: All I-MAESTRO Client Tools have a similar structure. Several different Client Tools will be studied and developed, for Students, Teachers, conductors, impaired Students, assessment experts, etc. They will support different devices: PC, Tablet and simpler PDA tools for sharing information and Lessons. (iii) I-MAESTRO Production Tools: Music Exercise Authoring Tool, Music Exercise Generators, etc. Repository t1 Arm posture and gesture Head posture and gesture Body to body gesture communication Bow gesture model creation Teacher Bow gesture consistency monitoring Gesture controlled creative multimedia interface Student Body Gesture and Posture Figure 2. Gesture and Posture Support 136

Page  00000137 When practicing using the above mentioned software modules, a student can view the posture and gesture sequences (3D rendering of the recorded motion data) as prepared by the teacher, selecting viewpoints and studying the recording without the limitations of a normal 2D video. A student can also make use of the I-MAESTRO systems to capture and study their own posture and gesture, or to compare them with some selected models. Current works in hand for I-MAESTRO gesture-based supporting tools can be categorised into (i) Instrument (bow and the main body of the instrument); (ii) Arm gesture (with respect to the instrument and upper body); (iii) Body (head and upper body); and (iv) Overall motion (including body to body gesture communication). The main functionalities of these tools are to provide automated analysis of the playing. For example, to analyse the regularity of a bow stroke exercise using the 3D motion data (see Figure 4). These tools are not to support direct "cloning" of the playing gesture since playing gesture is player dependent. They analyse the playing gesture to provide information on the relationship between the player and the instruments and the bow and the instruments, such as joint angles, space, flow, etc., acting as a 3D digital "enhanced mirror" that can reconstruct the playing to facilitate further the understanding of the relationship between physical and musical gesture (Ng 2005). I Figure 3(a). 3D rendering of recorded motion data Figure 4. A violin and bow with markers for 3D motion capture. Based on the gesture analysis, creative interfaces can be created by utilising several I-MAESTRO tools to provide gesture controlled performance for motivating music learning at early stages and also to support casual interests. For example, miming playing gesture to control an audio synthesiser to explore new tonality and new possibilities with digital media technologies; using gesture to control 3D graphics/animations with attractive visualisation, for edutainment purposes. Gesture mapping using these technologies can be adopted to map different input for multimedia controls to widen participation and accessibilities to all (Ng et al. 2005, Ng et al. 2004, Ng 2004, Traube et al. 2003). Since posture and gesture of musical performance contributes to various nature of communication and expression aspects, the I-MAESTRO posture and gesture support tools also aim to analyse the inter-communications of two players (in a chamber scenario) to understand performance gesture for synchronisation such as starting, stopping, phrasing, changes in dynamics and tempo. renuering o0 recorueu Imoion oata 137

Page  00000138 4 Conclusions This paper discussed the I-MAESTRO project and presented its basic aims and objectives. After a general overview of the project, the paper focused on the discussion on gesture-based analysis area, exploring a number of supporting tools to facilitate string instrument learning and teaching. I-MAESTRO utilises and improves the current standards related to music education. The project is exploring a unified educational model integrating theory and practise to maximise efficiency, motivation and interests in the learning processes. It is also focusing on the production of guidelines on how the music tuition courseware can be implemented in standard tools and models for distance learning. The outcomes will be validated through a set of tools and methodologies including (i) tools and methods for music courseware production; (ii) interactive and creative tools for instrumental training in different environments such as classrooms and cooperative work, on different devices such as PC and Tablet-PC; and (iii) assessment and pedagogical models for music training. The project includes demonstration and validation activities, which will be conducted by major European institutions such as Accademia Nazionale di Santa Cecilia (Rome), the Fundaci6n Alb6niz (Madrid) and IRCAM (Paris). The final results of the project will be incorporated in various products and services that will be made available to the public and to educational establishments. Learning, The Reflective Conservatoire, International Conference: Apprentices and Sorcerers, 16-18 February 2006 Barbican, London, UK Ng, K., Bernardini, N., Hunt, A., Arfib, D. 2005. COST287 -ConGAS: Gesture Controlled Audio System, in Proceedings of the 5th MUSICNETWORK Open Workshop, Universitat ftir Musik und darstellende Kunst Wien, Vienna, Austria Ng, K.C. 2005. 3D Gesture Interface and Analysis for Performing Arts, First London 3D Imaging Technology Conference, London, UK Ng, K.C. 2004. University of Leeds Interdisciplinary Centre for Scientific Research in Music, in Proceedings of the International Computer Music Conference (ICMC'04), Miami, USA Ng, K.C. (guest ed.). 2005a. Journal ofNew Music Research (JNMR) special issue on Multimedia Music and the World Wide Web, 34(2), ISSN: 0929-8215, Routledge Ng., K.C., Camurri, A., and Bernardini, N. (eds.). 2004. Proceedings of the Symposium on Gesture Interfaces for Multimedia Systems, AISB 2004 Convention: Motion, Emotion and Cognition, The Society for the Study of Artificial Intelligence and the Simulation of Behaviour (SSAISB), ISBN: 1 902956374 Ng, K.C. 2004. Music via Motion (MvM): Trans-Domain Mapping of Motion and Sound for Interactive Performances, in Gunnar Johannsen (ed.) Proceedings of the IEEE, Special Issue on: Engineering and Music - Supervisory Control & Auditory Communication, 92(4): 645 - 655. Traube, C., P. Depalle, and M. Wanderley. 2003. Indirect Acquisition of Instrumental Gesture Based on Signal, Physical and Perceptual Information. in Conference on New Interfaces for Musical Expression (NIME-03), Montreal, Canada 5 Acknowledgements The I-MAESTRO project is partially supported by the European Community under the Information Society Technologies (IST) priority of the 6th Framework Programme for R&D (IST-026883,, Thanks to all I-MAESTRO project partners and participants, for their interests, contributions and collaborations. 6 References Crombie, D., Fuschi, D., Mitolo, N., Nesi, P., Ng, K., and Ong, B. 2005. Bringing Music Industry into the Interactive Multimedia Age, 1st AXMEDIS International Conference, Florence, Italy Mitolo, N., Nesi, P., and Ng, K.C. (eds.). 2005. Proceedings of the 5th MUSICNETWORK Open Workshop, Universitat ftir Musik und darstellende Kunst Wien, Vienna, Austria Ng, K. (ed.) 2006. Proceedings of the 2nd International Symposium on Gesture Interface for Multimedia Systems (GIMS2006), ISBN 0 85316 246 8, 9-10 May 2006, Leeds, UK Ng, K. 2006. Creative Multimedia Interfaces and Collaborative Technology Enhanced Environment for Music Playing and 138