Page  00000001 Musical Activities at the Audiovisual Institute of the Pompeu Fabra University Xavier Serra Audiovisual Institute - Pompeu Fabra University Rambla 31, 08002 Barcelona, Spain Abstract The Audiovisual Institute ( is a research center of the Pompeu Fabra University, ( located in Barcelona and devoted to new media technologies. Apart from its main research activities, the Institute also promotes the production of experimental media art, being Music one its main focuses. In this report we present the main music research projects being carried out and related music activities of the Institute. 1. Introduction The Audiovisual Institute, IUA, was established in 1994 as a research center of the Pompeu Fabra University, UPF, linked to its School of Audiovisual Communication. In its six years of history, the IUA has evolved into an autonomous center of the university with links to several departments, with more than 50 members (including researchers, graduate students, technical and administrative staff), with most of its funding coming from outside the university, and with great emphasis on music research and production. The IUA is currently located in downtown Barcelona (La Rambla) and will move, in the fall of 2000, to a nearby site that is part of the old train station of the city (Estaci6 de Franga). This station is slowly being converted into a complex dedicated to education, research and production in the area of Communications. This new location will allow a further expansion of the Institute and its involvement in new initiatives. The aim of the IUA is to provide a suitable environment for research and experimentation aimed at students, professionals, the industry, and society at large. It is a meeting point for people coming from traditionally separated fields: science, art, engineering, design, computer science, communication, etc. Research at the IUA revolves around digital media, developing basic technologies and software tools for production, analyzing the new languages of interactive communication and studying its impact in the production and presentation of audiovisual works. The research is structured around several groups; each one dedicated to a different subject, being the Music Technology Group the larger one of them. Even though the IUA is not a university Department, most of its members are involved in different undergraduate and graduate programs of the UPF. At the undergraduate level the IUA participates in the Diploma in Computer Systems (specialized in Communication), in the program on Audiovisual Communication, and in the program on Computer Engineering (specialized in Communication). At the graduate level the IUA is involved in different post-graduate courses such as the Master in Digital Arts, the Doctoral program on Computer Science and Digital Communication and the Doctoral program on Social Communication. The production at the IUA is always looking for the innovative or experimental element in the new languages coming out from the use of digital media. The IUA has produced, in its six year of history, a large number artistic and cultural CD-Roms, Virtual Reality installations, music works, videos, Webs, etc. Other experimental productions are done within the framework of the Audiovisual and Musical Scholarships for Young Creators promoted by the IUA and the Phonos Foundation or within the different education programs of the university. Many dissemination activities are organized by the IUA and the Phonos Foundation, such as the Sala Hal, the Research or Production Seminars, Concerts Phonos, or the Tools for Composition courses. Next we describe each of the main activities of the IUA in more detail.

Page  00000002 2. Music and Audio Research The Music Technology Group (MTG) of the IUA ( has been mainly known for its research on digital sound modeling using spectral analysis/synthesis techniques. From that work, the approximate 18 members of the group are involved in various projects that expand that original effort into other related areas. The MTG is actively involved in a number of European projects and initiatives. Apart from the research projects described later, we participate in different research networks such as the Action-G6 "Digital Audio Effects" ( of the European COST program or in the newly created MOSART network dedicated to Sound and Music Computing and with the major goal of training young researchers. Next we will briefly describe the main research initiatives of the group. Spectral Modeling Synthesis We use the term SMS to describe our ongoing research on the development of techniques and software applications for the analysis, transformation and synthesis of musical sounds based on spectral models (Serra, 1997). In the past few years the research on SMS has been centered on the extraction of high-level sound attributes, that is, going from the original low-level deterministic plus stochastic representation to higher levels of abstraction (Serra and Bonada, 1998). We have also developed several software applications with graphical interfaces, for different research and musical needs (Resina and Serra, 1999), developed languages for its control (Amatriain, Bonada, Serra, 1998) and data formats for the storage and exchange of the analysis data (De Boer, Bonada, Serra, 2000). Information on SMS and public domain software applications based on it can be found in our Web site ( Content Processing of Audio Under the term Content Processing of Audio we include research projects focusing on sound analysis techniques that go beyond traditional signal processing approaches. These techniques try to solve problems such as signal understanding, symbolic representation, or even automatic semantic labeling of sounds. Some of the basic techniques are already quite well known in the Computer Music research field and in other disciplines such as speech recognition, but there is room for many new developments. In particular we are involved in three research projects, funded by the European Commission, related to the automatic extraction of sound descriptors for identifying, indexing and/or processing musical sounds. CUIDAD. ( This is a European Working Group coordinated by Ircam aimed at gathering all institutions, industrials and users interested in the content processing of music. As part of this project the IUA is actively involved in the Mpeg-7 standardization process (Herrera, Serra, Peeters, 1999; Peeters, McAdams, Herrera, 2000). RAA. ( This is a project of the European Information Society Technology (IST) initiative. The ambitious goal of RAA is the definition of a so-called AudioDNA. This AudioDNA, which is derived automatically from the audio-signal, uniquely describes an audio-title, like a human DNA describes the human being. Knowledge of a certain AudioDNA allows identifying the corresponding title within an unknown audio signal. The applications to be developed as part of the project will perform automatic identification of music titles on the Radio and Internet. The role of the IUA in the project is the definition and implementation of the core technology for the AudioDNA extraction and the title identification process. CUIDADO. This is also an IST project that starts this year and that can be considered a continuation of CUIDAD. This one aims at developing technologies for contentbased modules and applications using the MPEG7 Media representation standard. The project covers the analysis process (extraction of descriptors), the navigation process (retrieval methods and interfaces implemented in a database system with Web interoperability), and the creative process (consuming and authoring tools). The main involvement of the IUA is related to one of the applications to be developed in the project, the Sound Palette. This application is an authoring tool for retrieving, processing and editing sound fragments. This application will require the development of high- and low-level descriptors for rhythm, expressivity and sound object classification; the development of other descriptor extraction and processing modules relevant for the sound editing features of the Sound Palette; and the design of feasible content-based transformation procedures. Singing voice processing In collaboration with Yamaha we are developing analysis/synthesis technologies specific to the singing voice. One of the main developments has been a sound morphing system for karaoke applications (Bonada, Cano, Loscos, Serra, 2000; Bonada, Cano, Loscos, De Boer, Serra, 2000). This project involves the implementation of SMS as a real-time analysis-synthesis system and the realtime morphing of features of the user singer with the singer we want the user voice to resemble. This introduces an alignment problem that has to be solved in real-time. For that purpose speech recognition tools based on Hidden Markov Models have been integrated in the system and

Page  00000003 adapted to the peculiarities of the singing voice and realtime (Loscos, Cano, Bonada, 1999). On-line Sound The idea of a Web-based studio, a musical production environment based on the Web, has been an inspiring goal for several projects over the last few years (Loureiro, Serra, 1997). The main idea is to build a network application that takes advantage of the client-server architecture of the Web. A host machine is used as a sound database and a processing server and the control is in the client machines. A user can access and use the processing and storage potential of a remote studio in a friendly and interactive manner. Our work in progress on this project can be found in our Web site ( Modeling of expressive performances In collaboration with the Artificial Intelligence Research Institute (IIIA) of the Spanish Scientific Research Council (CSIC) we have started to study the issue of expressiveness in music interpretation. In particular we have worked on the extraction of musically meaningful sound attributes in the context of SMS and its use in the analysis/transformation/synthesis of music expression (Arcos, Lopez de Mintaras, Serra, 1998). Interactive Systems We are slowly starting to get involved in the design of interfaces and systems for interactive composition and performance. In the context of SMS we have developed a real-time performance system (Resina and Serra, 1999) and another project has been the development of FMOL (Jordh, 1999) a collective music composition application based on Internet using real-time synthesis and interactive interfaces. 3. Education The UPF is Catalonia's fourth publicly funded university. It is a young university, yet it already has its own little slice of history: it was founded in 1990 with 316 students taking two degree courses, and today it has around 9,000 students, fourteen degree courses and an expanding campus in the heart of the city, stretching between the Rambla and the district of Poblenou. Most of the people from the IUA participate, either in the School of Audiovisual Communication or in the Polytechnic School of the UPF. The School of Audiovisual Communication offers the undergraduate degree on Audiovisual Communication and the Doctoral program on Social Communication. The Polytechnic School offers the Diploma on Computer Systems (specialized in Communication), the undergraduate degree on Computer Engineering (specialized in Communication) and the Doctoral program on Computer Science and Digital Communication. The complementary approaches of these two Schools are ideal for the aims of the IUA. Independently of the university departments the IUA offers several post-graduate courses. The main one is the Master in Digital Arts, which is a one year program with a common core of subjects and two specialties: Image and Music. 4. Production Since its foundation the IUA has considered artistic production an important and necessary complement to the research and education activities. We have engaged in several big productions, like CD-Roms and Virtual Reality Installations and many music works, videos, etc.. Apart from the more institutional productions, the IUA promotes experimental creations by offering scholarships to young artists. The objective of the annual program offering scholarships for audiovisual and musical creation is to promote experimental audiovisuals (video art, animation, on line and off line multimedia, virtual environments, etc.) and musical productions (electroacoustic music, mixed music, electronic music, etc.) with the aid of computer-based systems. 5. Dissemination The most prominent dissemination activities of the IUA are: Sala Hal. Series of projections, conferences and concerts which show current experimental and creative audiovisuals, but also consider the forerunners of these genres in the history of cinema and music. Sala HAL focuses on alternative productions, either produced at the IUA or outside, whose originality stems from either the way in which they were created or their innovative character. Phonos Foundation. Since 1994 Phonos Foundation ( is part of the IUA. Phonos was established in 1975 and for many years it was the only laboratory of electroacoustic music in Spain. Through the courses given there passed most of the Spanish musicians that are currently working with electronic and software technologies. Phonos organizes concerts (the Phonos Concerts series has arrived this year to its 25th anniversary), conferences and other activities to spread both the works done in the IUA and in similar centers around the world. Phonos is supported by the main Spanish and Catalan cultural institutions. Research and Production Seminars. In the past few years the IUA has organized more than 200 lectures and

Page  00000004 seminars covering a wide spectrum of topics with the participation of invited lecturers from around the world. Publications. The IUA has been developing and promoting several on-line publications and web sites related with Audiovisual and Musical Creation; Communication; Interactive Literature, etc.. We are currently developing a Web-based Mediateque that will include a database containing most of the works and documents produced at the IUA, in a variety of multimedia formats and accessible on-line. All these publications are available in the IUA Web-site ( In collaboration with the School of Audiovisual Communication, the IUA publishes an electronic magazine, Formats ( whose priority is the presentation of research articles in the area of audiovisual communication. Another magazine that the IUA is collaborating with is CODEXXI ( an academic paper journal dedicated to the musical communication field. References Amatriain, X. Bonada, J. Serra, X. 1998. "METRIX: A Musical Data Definition Language and Data Structure for a Spectral Modeling Based Synthesizer". Proceedings of 98 Digital Audio Effects Workshop, Barcelona 1998. Arcos, J. L.; R. L6pez de Mintaras; X. Serra. 1998. "Saxex: a Case-Based Reasoning System for Generating Expressive Musical Performances", Journal of New Music Research, Vol. 27, N. 3, Sept. 1998. Bonada, J.; P. Cano; A. Loscos; M. de Boer; X. Serra. 2000. "Singing Voice Impersonator Application for PC", Proceedings of the ICMC2000. Bonada, J.; P. Cano; A. Loscos; X. Serra. 2000. "Voice Morphing System for Impersonating in Karaoke Applications". Proceedings of the ICMC2000. De Boer, M.; J. Bonada; X. Serra. 2000. "Using the Sound Description Interchange Format within the SMS Applications", Proceedings of the ICMC2000. Herrera, P.; J. Bonada. 1998. "Vibrato extraction and parameterization in the Spectral Modeling Synthesis framework". Proceedings of 98 Digital Audio Effects Workshop. Barcelona, 1998. Herrera, P.; X. Serra; G. Peeters. 1999. "Audio Descriptors and Descriptor Schemes in the Context of MPEG-7", Proceedings of the ICMC99. Jorda, S. 1999. "Faust Music On Line (FMOL): An Approach to Real-Time Collective Composition on the Internet", Leonardo Music Journal, vol. 9, 1999. Loscos, A.; E. Resina. 1998. "SmsPerformer: A RealTime Synthesis Interface for SMS". Proceedings of 98 Digital Audio Effects Workshop, Barcelona 1998. Loscos, A.; P. Cano; J.Bonada. 1999. "Low-Delay Singing Voice Alignment to Text". Proceedings of ICMC99. Loureiro, R.; X. Serra. 1997. "A Web Interface for a Sound Database and Processing System", Proceedings of the ICMC97. Peeters, G.; S. McAdams; P. Herrera. 2000. "Instrument Sound Description in the Context of MPEG-7", Proceedings of the ICMC2000. Resina, E. 1998. "SMS Composer and SMS Conductor: Applications for Spectral Modeling Synthesis Composition and Performance". Proceedings of 98 Digital Audio Effects Workshop, Barcelona 1998. Resina, E.; X. Serra. 1999. "The Musicians' Software Mall: A Set of Composition and Performance Oriented Applications for Sound Synthesis", Proceedings of the ICMC 1999. Serra, X.; J. Bonada. 1998. "Sound Transformations on the SMS High Level Attributes". Proceedings of 98 Digital Audio Effects Workshop, Barcelona 1998. Serra, X. 1997. "Musical Sound Modeling with Sinusoids plus Noise". G. D. Poli and others (eds.), Musical Signal Processing, Swets & Zeitlinger Publishers, 1997.