Page  00000001 TOWARDS A ROADMAP FOR THE RESEARCH IN MUSIC TECHNOLOGY Xavier Serra Music Technology Group, Audiovisual Institute Universitat Pompeu Fabra Barcelona, Spain ABSTRACT In this article I give a personal view on what could be a roadmap for the research in Music Technology. I will describe the context of this research, the current state of the art, the challenges that lie ahead and I will conclude with some strategies to face those challenges. Given that to write a roadmap for a given discipline is a very challenging task that can only be the result of wide discussion within its research community with this article my only goal is to help start that discussion. 1. INTRODUCTION Given the nature of Music Technology there have been very few initiatives to promote large collaborative projects that could tackle some of the big issues that are relevant in our field, issues that would benefit from this type of large initiatives. Most of the research is carried out by individual researchers or by small groups and there are few large projects, mainly due to the funding structures that our research is based on and that it has access to. Everyone would agree that our research field would greatly benefit from being able to carry out big collaborative projects and from sharing common methodologies and objectives. Most scientific and technological fields have quite clear identities and their researchers share methodologies and have a common understanding of the challenges that need to be confronted. In interdisciplinary fields, like Music Technology, this is much harder but the research communities of some of the most consolidated interdisciplinary fields are starting to promote roadmap initiatives to develop common research strategies [1] [2]. A research roadmap should be able to answer the following questions: * What are today's scientific and technological challenges? * What are the roadblocks to progress? * What do we need to do to overcome roadblocks? * What can't be accomplished by any single institution but is the responsibility of a given research community as a whole? Thus, a research roadmap in Music Technology should promote a common research agenda while maintaining the diversity of the approaches that coexists in our field and that are the key to our personality. In this article I just give my personal contribution to this discussion, being conscious that it lacks some major components, such as an extensive discussion of the state of the art and also a detailed proposal for actions to be carried out by the community. I have also avoided proposing a definition for Music Technology and a description of its research focus. I leave all this for the real roadmap. 2. MUSIC TECHNOLOGY AS AN INTERDISCIPLINARY RESEARCH FIELD Music Technology (MT), or the more restricted concept of Computer Music (CM), is a clear example of an interdisciplinary research field that sits at the cross road of many disciplines, such as Musicology, Acoustics, Signal Processing, Psychoacoustics, Artificial Intelligence, and Human-Computer Interaction. An important consequence of this interdisciplinarity is that it affects the methodology to be used, since at the same time different methodologies coexist that come from the different research traditions of science, technology and social sciences. There are quite a number of research areas that are the result of interdisciplinary work and there is quite a lot of discussion on the nature of this type of disciplines [3] [4] [5]. There are also criteria for evaluating the research being produced in these fields. For example the following are three criteria that this type of research should conform to: * Consistency with multiple separate disciplinary antecedents. the way in which the work stands vis 'a vis what researchers know and find tenable in the disciplines involved. * Balance in weaving together perspectives. the way in which the work stands together as a generative and coherent whole. * Effectiveness in advancing understanding. the way in which the integration advances the goals that researchers set for their pursuits and the methods they use. 3. INTERNATIONAL CONTEXT There are many researchers active in MT at the international level. A rough estimate of the number of researchers (only counting doctors or senior researchers) which consider MT the main focus of their research would be around 1000 people. This number is a rough estimate based on the attendance to the main yearly MT conferences. In this estimate we do not count the researchers that are only active in specific fields

Page  00000002 such as musical acoustics, audio engineering... and that do not interact with the rest of the MT community. Despite the large number of researchers, there are only a few research centers having MT as their main activity and covering both artistic and scientific/engineering topics. We could mention: IRCAM (Paris), CCRMA-Stanford (California), SARCQueens University (Belfast), CIRMMT-McGill University (Montreal), MTG-Pompeu Fabra University (Barcelona),... Most of the MT research groups belong to centers with a broader focus or with a focus different than MT. We could group the research groups in two categories depending on whether their output is mainly artist or scientific/technological. Examples of the ones with an artistic focus are: CRCA-UCSD (USA), GRM (Paris), Institute of Sonology (The Hague), NoTAM (Oslo), Grame (Lyon),.... and examples of the ones with a scientific/technological focus are: MIT-MediaLab (Massachusetts), HUT (Helsinki), Center for Digital Music - Queen Mary (London), InfoMus Lab - University of Genova, TMH-KTH (Stockholm),... The scientific/technological results of MT research are regularly presented in a variety of academic conferences, such as ICASSP and other IEEE conferences, AES conferences/conventions, ACM conferences, EUSIPCO, ASA meetings, ICAD,... The ICMC, started in 1974, was the first conference dedicated to CM and is still the only conference covering most aspects related to MT. More recently a number of conferences have been created to deal with some specific MT issues, such as: DAFX (digital audio effects), ISMIR (music information retrieval), or NIME (music interfaces) and they have been instrumental in gathering MT researchers with specific interests. There are also some relevant regional conferences. In terms of journals, the scientific/engineering results of MT are presented in a variety of scientific and engineering journals, such as JASA, J-AES, IEEE journals, ACM journals,... The Computer Music Journal, started in 1977, was the first one to be focused just on CM. More recently journals such as the JNMR or Organized Sound are also quite focused on the field of CM. Most of the MT researchers belong to one or several of the following professional associations: ICMA, AES, ASA, or IEEE. Not being an established and basic research discipline, MT does not have stable funding sources. It got some big grants from the US government in the 70s and 80s that helped the establishment of the field and in the late 90s there was some important funding from the EU that helped support and expand several European groups. But at this time there is no clear interest by the funding bodies to support in a direct way the MT research. In order to "survive", many research groups are "selling" their activity as part of broader or different topics. Culturally speaking, the CM research was originally quite close to the Contemporary Music culture. In fact, for most people, CM is synonymous of contemporary music. In recent years this has changed quite a bit and the aesthetic considerations are quite independent from the CM research. In fact this is why we tend to use MT instead of CM, to avoid the contemporary music connotation. MT has always been very much driven by technology advances and research developments coming from other fields. Thus, by following the technology waves it is possible to get an idea of the MT problems and applications that will be worked on in the next few years. MT has always been an applied field and thus it has always been quite close to the music industry. Originally, the "music industry" was mainly limited to the synthesizer manufacturers, but at this time the concept of music industry is not that clear anymore and there is a wide range of industrial sectors producing music related commercial products. Most of these industries sit at the nexus of cultural, entertainment, leisure and fast moving consumer goods industries. It is therefore in constant change. This industrial sector has been mainly composed of SMEs with very little research potential, thus the economic support for research and research centers has been small. Only a few companies have had long lasting join research projects with universities. In the last few years the growth of the MIR field has resulted in the creation of quite a number of small companies, or divisions within large companies, that produce and sell products and services related to MIR technologies. It is difficult to get a clear picture of all these industrial developments. Related with all these industrial changes we could now consider that the music industry is really part of what is called the Information and Communication Technologies sector, ICT. Thus by studying this sector we can try to understand our industrial context. The environment for the Information and Communication Technologies is changing drastically at the international level. According to the EU analysis documents: * ICT research is increasingly organized on an international scale, as firms seek to relocate their R&D activities in the face of accelerating competition in global markets, * innovation processes are more open, with wider and faster exchange of ideas, people and resources, * technology chains are increasingly complex, making it more difficult for any single player to establish industrial leadership in any ICT field, * new promising fields are emerging at the cross-over between ICT and other disciplines such as biotechnologies, materials and cognitive sciences.

Page  00000003 4. STATE OF THE ART IN MUSIC TECHNOLOGY RESEARCH There are many active research topics in Music Technology. From the first musical experiments with computers in the late fifties to the current work, the focus of the field has expanded enormously. At this time a meaningful grouping and short description of the active research could be the following. 4.1. Music Generation/Modeling This is mainly related to what is known as Algorithmic Composition and it includes an important artistic component. Because of the artistic aspect, it makes it hard to apply a systematic analysis of what is the state of the art and also what are the main research advancements. There were some important developments in the 70s and 80s but nothing much new has happened in the last decade. This research area has raised very interesting musicological (music analysis), mathematical (generation models) and software engineering (software languages) issues. At this time there are plenty of approaches and systems to automate the composition process and the driving force of this area is mainly within the context of the contemporary music tradition. 4.2. Sound Generation/Modeling This has been the major research area for the past 40 years and its practical applications have been centered on building sound synthesizers and audio processors. In the 90s the main focus was on Physical and Spectral Models, but despite all the good results obtained, most of the commercial synthesizers are still based on sampling and most of the audio processors are still based on time-domain techniques. The sound quality of the synthesizers based on physical models is not yet comparable to the samplers, and despite having other advantages, synthesizers based on Physical Models are not succeeding in the market place. Spectral Modeling has succeeded in being competitive in voice synthesizers, both for speech and singing voice, but not in the synthesis of the rest of the musical instruments. There are still quite a number of signal processing issues and other problems related to particular applications to be worked on. There is also a clear need for working on the problem of control and expressiveness of the synthesizers. But, in parallel, new paradigms for making sounds are starting to be developed, one is based on hybrid techniques, for example the idea of physically inspired synthesis and another one is based on feature-driven audio synthesis. Also known as Audio Mosaicing this last idea opens up a new way of thinking about synthesis. From the basic idea of automatically assembling audio segments using their automatically generated description a whole bunch of possibilities appear. Thus all this should be explored further. 4.3. Music Performance Analysis/Synthesis Despite being the performer a key element in the music making chain, there has been little formalization, or scientific studies, of what the performer really brings into that chain. Performance skills are learned in an intuitive way and thus they are quite elusive to scientific analysis. However in the past few years it has turned into a very active and fruitful field of research. Several approaches are now used for analyzing and modeling the performance component of music. Most of the current research is based on either measuring actual performances, applying analysis-by-synthesis methods, or developing performance models with machine learning methods. We are starting to be able to capture the differences between different performers, we now have models for capturing the mood in a given musical style, we are able to measure some expressive features from an audio recording of a performance, and we are also able to generate some aspects of an expressive performance. The practical applications of all this research might not be that far. 4.4. Music Interfaces In the context of electronic instruments it is feasible to separate the controlling aspect of a musical instrument from its sound producing capabilities. We can build controllers and interfaces to capture performance gestures and sound modules to produce sounds. With this division a staggering range of possibilities have become available. Since the invention of the first electronic instruments there has been considerable research on developing new controllers with which to explore new creative possibilities and communication protocols to interface the controllers with the sound generation devices. Thus, the concept of performance takes a new meaning and with it the concept of instrument. There is a huge open ground for research by considering the performer-instrument interface in the general framework of human-computer interaction. This is a research area that also has a very strong musical component and for which it is not easy to establish the criteria for defining progress. Some of the active research topics are the study of musical mapping algorithms, development of intelligent controllers and of novel controllers for collaborative performance. A clear example of the paradigm shift that is taking place is related with Internet. Internet is becoming a medium with which to make music and thus a new musical interface to be explored. 4.5. Music and Audio Understanding/Retrieval This is currently the most active research area in Music Technology. We are far from understanding the relationship between acoustics, music structure, cognition, and emotion, but these relationships offer a great ground for scientific and technological research.

Page  00000004 There are a lot of issues involved in the process that goes from the sound waveform entering our ear to the music sensation we get from that sound. We understand some of the low level (hearing system) issues but we are far from understanding the cognition questions. Currently we have good insights on aspects of auditory perception, some of which are: basic auditory processes, low-high grouping mechanisms, timbre, pitch, time and rhythm perception. Beyond this low level there are many open research topics related to disciplines such as psychoacoustics, psychology, music perception and cognition, neuroscience... By using signal processing and machine learning techniques we have been able to extract and identify meaningful and useful information from musical signals. This automatically extracted information can be used in many of the problems that are currently worked on within the Music Information Retrieval area. We are starting to have results that can be used in practical applications and the progress is so fast that it is difficult to predict what will happen in the next few years. 5. MAJOR CHALLENGES RELATED TO MUSIC TECHNOLOGY RESEARCH Using the ideas that have shaped the current EU research funding programs we can get some useful pointers for identifying the challenges that we will have ahead of us, both in terms of research and in terms of technology transfer. Here are some of the EU strategic objectives that are more related to Music Technology: * Networked audiovisual systems and home platforms. "Audio Visual" systems and applications in converged and interoperable environments encompassing broadcasting, communications, mobility and IP. * Semantic-based knowledge systems. semanticbased and context-aware systems to acquire, organize, process, share and use the knowledge embedded in multimedia content. * Cognitive systems. artificial systems that can interpret data arising from real-world events and processes (mainly in the form of data-streams from sensors of all types and in particular from visual and/or audio sources); acquire situated knowledge of their environment; act, make or suggest decisions and communicate with people on human terms, thereby supporting them in performing complex tasks. * Technology-enhanced access to cultural heritage: systems and tools which will support the accessibility and use over time of digital cultural and scientific resources. This requires work to: (1) Support the emerging complexity of digital cultural and scientific objects and repositories, through enriched conceptual representations, and advanced access methods, (2) Explore how to preserve the availability of digital resources over time, through novel concepts, techniques and tools. * Cross-media content for leisure and entertainment. the full digital content chain, covering creation, acquisition, management and production, through effective multimedia technologies enabling multichannel, cross-platform access to media, entertainment and leisure content in the form of film, music, games, news and alike. 6. SOME FUTURE STRATEGIES It is really beyond the aim of this article to make concrete proposals for the advancement of our research field. But I would just like to finish it with some ideas for organizational and scientific strategies that could be considered in the roadmap. On the organizational side there is a clear need for: (1) More interdisciplinary research projects covering both artistic and technical work, (2) More collaborative projects between research centers, (3) Stronger cooperation between academic researchers and industrial companies, (4) Better common grounds for comparing research results, (5) Stronger collaboration with other fields of research, (6) More rigorous assessment of MT research results, and (7) Better organizational structures of the MT research community. On the scientific side there is a clear need for: (1) More emphasis on artistic and aesthetic issues, (2) More weight on the incorporation of the new scientific developments from fields such as neuroscience, databases, or human-machine interaction, (3) More work on the problem of control and expressiveness of sound synthesizers, (4) More work on new synthesis techniques based on hybrid techniques and feature driven synthesis, and (5) more emphasis on collaborative performance systems. 7. REFERENCES [1] ECVision. 2005. A Research Roadmap of Cognitive Vision. Version 4.2. [2] Finkelstein, A. & J. Kramer. 2000. Software Engineering: A Roadmap. Proceedings of the Conference on The Future of Software Engineering. [3] Metzger, N. & R. Zare. 1999. Science Policy: Interdisciplinary Research: From Belief to Reality. Science 1999 283: 642-643. [4] Chubin, DE.; AL. Porter, FA. Rossini. 1984. An approach to characterizing interdisciplinary research. Journal of the American Society for Information Science. [5] Bruhn JG. 2000. Interdisciplinary research: a philosophy, art form, artifact or antidote?. Integr Physiol Behav Sci. 2000 Jan-Mar;35(1):58-66.