Page  00000001 Distributed M~usical Rehearsal Dimitri Konstantas (1) Yann Orlarey (2) Simon Gibbs (3) Olivier Carbonel (2) (1) University of Geneva - GUI, 24 rue General Dufour, CH-121 1 Geneva, Switzerland, Dimitri.Konstantas ecui.unige.ch (2) GRAME, 6, quai Jean Moulin, F-69202 Lyon, France, orlarey @grame.fr, carbo @rd.grame.fr (3) GMD, IMK.VMSD, Digital Media Lab, Schloss Birlinghoven, D-53724 Sankt Augustin, Germany, Simon.Gibbs @gmd.de Abstract Bringing together a group of musicians and a conductor for a musical rehearsal requires advance planning and an important budget in order to cover travel and subsidiary costs. We have developed an ATML~ based telepresence environment allowing small groups of musicians that are located in different sites to rehearse as if they were present in the same room and have organized a first distributed musical rehearsal trial. The trial allowed us to test and evaluate the system, according to a methodology we developed and to draw first conclusions regarding its performance and usability. 1 Introduction The rapid advances in telecommunication technologies and the wide availability of communication networks have allowed the introduction during the last few years of several services and applications for remote human contact and communication. Services like tele-presence, teleconferencing and video-phone as well as access to medium bandwidth communication networks (JSDN) are now available not only to large industrial companies, but also for private home usage. The ultimate goal of these services and applications is to allow people that are situated in remote sites to talk, work and interact as if they were phy'sically present in the same room. During the last decade applications achieving partially this goal were restricted to a very small number of special users, using private communication networks and expensive technological infrastructure (in the order of millions ECUs). It is only during the last couple of years with the introduction of high bandwidth terrestrial public networks like ATM and the appearance in the market of moderately priced communication and telepresence equipment that telepresence services are becoming available to a larger public. This work was supported by the Commission of the European Union in the frame of the ACTS project DVP (AC 089) and by the Swiss Federal Government with the OFES grant No. 95.0493 One of the goals of the Distributed Video Production (DVP) [1] project is to evaluate state of the art communication and telepresence equipment for innovative distributed applications. Towards this goal a telepresence environment for the support of Distributed Rehearsals has been developed [2]. The aim of the distributed rehearsal application is to allow the organization of rehearsals (theatrical, musical etc.) without requiring the participants (actors, musicians, conductors etc.) to be physically in the same room, eliminating thus the need to travel from one city to another for participating in the rehearsal. From the different types of rehearsals we choose the M~usical Rehearsal as a pilot application. The reason is that the audio and video requirements for a musical rehearsal are much higher than other types of rehearsals and we are thus able to test the distributed rehearsal environment in a worst case situation and study the present technology limits of telepresence applications. Our work in the development of the distributed musical rehearsal environment follows two major paths.The first is the technical path and the second is the artistic path. In the technical path all the issues related to the development of the environment, like the measurement and reduction of the encoding and transmission delays, the measurement and improvement of the sound and video quality, the establishment of the communication lines etc. are studied and different techniques are tested. The artistic path on the other hand concentrates in the organization and evaluation of the musical rehearsals, and namely in

Page  00000002 the selection of the musical pieces, the organization of the orchestra and most importantly the development of a methodology for the objective measurement of the quality of the distributed rehearsal environment. In this paper we give an overview of telepresence environment we have developed and present our first results from the distributed rehearsal trial organized in November 1,5th 1996, where the GRAME-BOC music ensemble, performed a whole day distributed rehearsal between the University of Geneva and GMD, the German National Research Center for Information Technology, located outside Bonn. 2 Technical Infrastructure The technical requirements for organizing a distributed rehearsal are basically these of a telepresence session. That means the existence of (at least) a video wall [3] with minimal dimensions 2x,3 meters with good quality video projection, and a hi-fl audio system. The goal is to give the impression to the participants that they are physically in the same room. However for conducting a distributed musical rehearsal these requirements are not enough. Conducting an orchestra is done using body language. The conductor coordinates the musicians with gestures, eye contact, body expressions etc. As a result the requirements of the telepresence environment for a distributed rehearsal are much higher than normal situations. The video quality should be high enough to allow musicians to correctly see and interpret the conductor' s signals and be able to see even where the conductor is watching. The conductor on the other hand should be able to identify correctly the position of each musician, as if they were physically in the same room. This requires both a high quality video and an accurate,3D restitution of the sound. In addition the delay from the moment the conductor gives a signal until the moment he perceives the response of the musicians to his signal should be comparable to the delay of a normal rehearsal. This means that the delay must be less than 100 ins, which is roughly the equivalent of a distance of 30 meters between the conductor and the musicians. The most important technical issues to face in a distributed rehearsal system are how to minimize the total transmission delay and how to accurately reproduce the image and the sound space. The element contributing the most in the transmission delay is the digital encoding of the video. State of the art MPEG encoders require at least SO00ms for the encoding, which is problematic even for teleconferencing applications. It was for this reason that we choose MJPEG encoding where the encoding and ATM packetization does not exceed 2 frames (80 ins). Of course someone may claim that the use of analogue video via analogue lines does not have this limitation. However analogue video requires higher bandwidth communication lines (at least 60 Mbps), it can only be used in a oneto-one connection, and it cannot be easily processed for digital effects (like virtual studio technics, 3D rendering, etc. [4]). For the transmission of the encoded video and audio we used a 24 Mbps ATM line connection between the University of Geneva and the GMD headquarters. The accurate reproduction of the,3D sound space requires, naturally, an accurate capture of the sound. Different technologies are available for this: multiple microphones connected to a matrix, spacializers, dummy head etc. In the musical rehearsal trial we experimented with multiple microphones and a dummy head, for capturing the sound at the orchestra site. The dummy head was placed approximately where the conductor's head would be if he was physically present in the rehearsal. The dual microphone was placed next to the video wall. For both systems a matrix was used for the correct three dimensional reproduction of the sound [,5]. For the reproduction we tested both a non-intrusive and an intrusive system. Namely the conductor was listening to the sound from loud speakers (non-intrusive system) and using a head-set (intrusive system). 3 The Distributed Rehearsal Trial The piece retained for the distributed musical rehearsal trial, "De'rive", was composed in 1984 by Pierre Boulez for six instruments: piano, vibraphone, violin, cello, flute and clarinet. For the trial the six musicians were installed in CUI - Geneva and the conductor was in GMD - Bonn. The total duration of the rehearsal was 6 hours and it was organized into four phases. The first phase consisted of the tuning of the equipment and positioning the dummy head and the dual microphone, for the correct capture and reproduction of the three dimensional sound space. The second phase was dedicated in the quantitative quality control of the installation, which we describe in more detail in the next section. The third phase was the main part of the trial where the orchestra rehearsed the retained musical piece. It was organized into two sessions: in the first session the sound was reproduced at the conductors site using loud-speakers placed in well defined positions for the correct reproduction of the sound space, while in the second session the dummy head was used and the conductor used a head set for listening to the orchestra.

Page  00000003 The trial was completed with a set of interviews taken by the test evaluator from all the musicians and the conductor. This allowed the participants to state their personal opinion and feeling of the experiment and obtain subjective measurements of the quality of the trial. 4 Evaluation Methodology & Results A very important issue for the distributed rehearsal system is the measurement of its quality and in consequence the limits of its usability. The quality of the system can be measured subjectively and objectively. Although subjective measurements are very important for the users of the system, we needed objective measurements in order to evaluate different options and technology choices. For this reason we developed within the project a methodology for the objective measurement of the system quality [6]. The methodology is based on the fact that in a musical rehearsal the conductor must control, identify and possibly modify notes that are coming wrong in the musicians' scores. During the second phase of the trial specific scores were given to the musicians and the conductor. The scores given to the musicians contained various errors (like time errors, pitch errors, dynamic errors etc.) when compared to the score given to the conductor. The errors ranged from very easy to detect, to very difficult ones. The conductor was then asked to detect the errors in the musicians scores. By reproducing the same test in a local situation with different but equivalent test scores, and comparing the errors found in the distributed rehearsal and in the local rehearsal we will be able to establish a concrete measurement of the system quality. For example for a specific set of scores (Varia Rouge) Table 1 gives the distribution of the errors detected per category and Table 2 the errors detected per instrument. In order to have more complete results the musicians and conductor were interviewed independently after the trial. A detailed analysis of the interviews, and the interactions and discussions between the conductor and musicians during the trial, was also performed by a psychologist. This allow us to take into account the change in behavior of the participants and the different ways they are adapting to different problems and differences between the local and the distributed situation. For example Table 3 gives some of the negative and positive points extracted from the conductor's interview. At the time of the writing of this paper we had not yet reproduced the local situation. As a result we can draw no definite conclusions from the measurements obtained and analyzed for the distributed situation. Nevertheless some Category Errors Detected % Tone 8 4 50% Rhythm 5 0 0% Harmony 6 1 17% Nuances 3 2 67% Table 1 Errors per category Instrument Errors Detected % Vibraphone 5 2 40% Flute 1 0 0% Clarinet 6 3 50% Piano 4 1 25% Violin 2 1 50% Cello 4 0 0% Table 2 Errors per instrument Negative Positive Adjustment according to my Modest annoyance regardacoustic image ing what I was expecting Refine the acoustic image It seems to work Refusal of visual adjustment, The exchange of nice words bad image is inevitable I did notfeel the absorption Modest video delay, modest of energy that I give annoyance Dark image We found the good tone Refusal to create a visual We advanced in the intercommunication pretation of the piece Working space not so comfortable Open working space Analysis of sound sources Table 3 Negative and positive points of the distributed situation for the conductor. subjective conclusions can be drawn based on the comments of the musicians and conductor. First of all it is interesting to notice that when the musicians first came in the studio in order to try the distributed rehearsal system, they had reservations and doubts both for the principle of conducting a distributed rehearsal and for the capabilities of the technology. However after the first trial was completed they were convinced first on the principle and usefulness of conducting a distributed rehearsal and second on the capabilities of the technology. The musicians also observed among others that the continuous attention to the projected video was tiring them and that the flatness of the screen did not allow then to accurately identify

Page  00000004 where the conductor was pointing. Another interesting observation was also that the musicians and the conductor preferred to have a smaller delay of the audio (20ms) than to synchronize it with the video which had a much longer delay (80ms). 5 Future work The target of a musical rehearsal is to prepare the musicians and the conductor for the final performance. This preparation requires a number of sessions (rehearsals), depending on the music to be performed, the expertise of the musicians and the conductor, etc. However bringing together a group of 10 to 20 musicians and a conductor for even 5 rehearsals requires advance planning (8 to 12 months or even more) and an important budget in order to cover travel and subsidiary expenses. In the frame of the DVP project we have developed a telepresence environment allowing small groups of musicians that are located in different sites to rehearse as if they were present in the same room. The system is based on video wall technology and digital video and audio transmission over ATM lines. A first full day distributed rehearsal trial was organized in November 15th 1996 between the University of Geneva and GMD in Bonn. The trial allowed us to test and evaluate the system, according to a methodology developed in the project. The first results of the trial evaluation are very encouraging and indicate that the system can be used for performing distributed musical rehearsals over long distances.This was demonstrated also by the fact that the rehearsed piece was later performed successfully in a concert without any other rehearsals. However we need yet to complete the experimentation in order to define up to which level of rehearsals the system can be used. Our work continues with target the improvement of the distributed rehearsal system. The main points we will concentrate are the 3D sound capture and reproduction and the transmission delay minimization. In order to improve the 3D sound space reproduction at the conductor's site, we will perform tests tracking the position of the conductor's head and adjusting accordingly the artificial head's sound capture. This can be done either by using a servo mechanism for turning the artificial head accordingly or with a spacializer [7] or even both. Experiments will also be performed for improving the perspective of the projected video image for both the musicians and the conductor. In the future we plan to organize similar musical rehearsal trials over ATM and refine the system correcting technical problems and limitations that will be identified during these trial. Finally in addition to the trials, we will organize some public events for the presentation of the technology and the related issues. This will be done in relation with our plans for the commercialization of the system and the possibility to install it in different music conservatories. 6 Acknowledgments The authors would like to acknowledge the help of the people that worked in setting up the experiments and namely the conductor Daniel Kawka, the musicians of the GRAME-EOC, Fabrice Philippe, Bernard Gaviot-Blanc, Henri-Charles Caget, Roland Meillier, Antoinette Lecampion and Stephane Genay, the test scores' composer Jerome Dorival, and the technical/scientific collaborators Costas Arapis, Laurent Dami, Thomi Pilioura, Jacques Flumet (CUI), Christian Breitender, Vali Lalioti, Lother Zier (GMD) and Michel Stievenard (GRAME). References [1] Distributed Video Production - DVP, ACTS project AC 089, http://www.gmd.de/DVP [2] Dimitri Konstantas, "A telepresence environment for the organization of distributed musical rehearsals", in Multimedia Objects, ed D. Tsichritizis, Centre Universitaire d'Informatique, Univer. of Geneva, 1997 [3] Chistian Breiteneder, Simon Gibbs and Constantin Arapis. "TELEPORT - An Augmented Reality Teleconferencing Environment", 3rd Eurographics Workshop on Virtual Environments, Monte Carlo, Monaco, February 1996. [4] S. Gibbs, C. Arapis, C. Breiteneder, V. Lalioti, S. Mostafawy, J. Speier, "Virtual Studios: The State of the Art", Eurographics'96 STAR Reports. [5] Michael Williams, "Enregistrement et Reproduction de l'Environnement Sonore Naturel", Proceedings of the Rencontres Musicales Pluridisciplinaires Informatique et Musique - Le Son & l'Espace, Ed. GRAME, March 31 - April 1, 1995, Lyon, France. [6] Yann Orlarey & Olivier Carbonel, "Evaluation d'un dispositif de re6ptition a distance: rapport interm6diare", GRAME Technical Report, 1997. [7] Jean-Marc Jot and Olivier Warusfel, "La Spatialisateur", Proceedings of the Rencontres Musicales Pluridisciplinaires Informatique et Musique - Le Son & l'Espace, Ed. GRAME, March 31 - April 1, 1995, Lyon, France.