Page  00000001 Music Informatics Laboratory Jens Arnspang, Kristoffer Jensen, Declan Murphy, Tue Haste Andersen and Georgios Marentakis Department of Datalogy, University of Copenhagen email: arnspang@diku.dk Abstract The first Music Informatics Laboratory at a Computer Science Department in Scandinavia has been formed and is active with research and education of Master Thesis students, Ph.D. Students and post Doc's. Our laboratory runs a European Network among 13 universities, the MOSART project (Music Orchestration Systems in Algorithmic Research and Technology). The network is coordinated by Jens Arnspang for Brussels, and includes a 1.300.000 Euro grant. The Music Informatics Laboratory is in close contact with industry in Denmark, in particular in effects and music representation, in score handling and score scanning tools, and in audio based games. The laboratory has a growing contact net within culture and performing art as well. 1 Motivation The motivation for the Musical Informatics Laboratory lies in the cross section of the fields computer science, computational acoustics and music. The latter have been studied for thousands of years, and a natural understanding of instruments in terms of their audible capabilities and treatment in playing situations have been obtained in a humaniora sense. Although already Pythagoras and other ancient scientists studied for example the relation between tones and strings, it is only in recent centuries, that physicists have grasped the nature of behaviour of tones and of sensations of tones. For example the peers Helmholtz and Rayleigh wrote about 'Sensations of Tones' and of 'The Theory of Sound' only a century ago as one of the first - and still best - treatments of characteristics of tones within a science of nature understanding. In this last century the field of acoustics have adopted the strive for understanding the production of tones, the transition between tones and the physical processes and parameters, that might be used to model 'timbre', 'pitch', 'tone transition' and other features of instruments, as known in a humaniora and practical sense within music performance. In recent decades the possibility of analysing such 'musical data' has become available through the calculating power of the computers, that virtually may provide a breakthrough in both analysis of musical patterns and in resynthesis of audible musical signals, derived from models, build by the analysis. At the same time, computer science as such is interested in describing, modelling and coding various sort of data, text, images, sound and music, both from a general viewpoint of data representation and data handling, and from specialised and application oriented viewpoints. One such application is computer music. As a field, Musical Informatics is beyond doubt interesting to science as well as society, just think of the numerous 'musical machines', available as commercial products. It is about time computer science deal with the further development of these and the very basic science of nature and psychophysic aspects of music data description, analysis and synthesis. 2 Objectives The following six major tasks have been identified as the objectives for the Music Informatics Laboratory: 1: 2: 3: 4: 5: 6: Identification and Representation of Timbre Space Control and Virtualisation of Instruments Scale Space Processing of Musical Signals Symbolic Recognition of Musical Patterns Computer Music Composition Tools Interactive Performance Systems Apart from task three above, these objectives are almost identical to those of the European research network MOSART (Mosart 2000), described below in the next section. In addition to the MOSART tasks our laboratory would in task three above like to address the possible relevance and use of scale space theory, as applied to musical signals. Scale Space techniques are very well known within image processing (Lindeberg 1996). (Jensen 1999a) used scalespace techniques to detect attack and release in musical

Page  00000002 isolated sounds. Scale-space techniques are rarely used in acoustics, audio signal processing or Music informatics. It is, however, a fact, that strong theorems exist in scale space theory, for example the top point theorem (Johansen 1994), allowing a complete representation by a continuos band limited signal from 'top points in scale space' only, and vice versa, a complete reconstruction (up to a scale factor) of the original signal is possible from these top points. Such a representation of audio signals allow a revisit of very basic questions, concerning characterisation and manipulation of timbre, of encoding and transmission of sound. Our group in Copenhagen is planning special projects in looking for answers to classic problems in musical sound analysis, using these new scale space techniques. 3 Research Network We are currently building a network of European contacts. We have defined a joint three-year project with the acronym MOSART (Mosart 2000), for which we have obtained a 1.300.000 Euro grant under the EU TMR/IHP Program. DIKU (our node) is the coordinator of this network. The project is concerned with Musical Orchestration Systems in Algorithmic Research and Technology. The MOSART Research Network Project The project was defined by our laboratory and the colleagues below in order to form a European network in the topics. There were limits to the size of the net, and obviously many more laboratories ought to be connected in such a network, allowing young researchers to participate in the rapidly growing field of Music Informatics. We most kindly invite other research entities interested for further co-operation beyond the present network for a discussion. The Principal Contractor of the network is University of Copenhagen, Denmark, Department of Datalogy. The other members of the network are, in a approximative North South order: 1) Norwegian University of Science and Technology, Trondheim, Norway, Department of Telecommunication, Acoustics Group 2) Royal Institute of Technology, Stockholm, Sweden, Department of Speech Music and Hearing 3) University of Arhus, Denmark, Computer Science Department 4) Danish Technical University, Lundtofte, Denmark, Department of Acoustic Technology 5) The University of Sheffield, United Kingdom, Department of Information Studies 6) University of Nijmegen, The Netherlands, Music Mind Machine Group 7) Austrian Research Institute for Artificial Intelligence, Wienna, Austria 8) Universita' degli Studi di Padova, Italia, Dipartimento di Electronica e Informatica 9) University of Genova, Italy, Dipartimento di Informatica Sistemistica e Telematica 10) Centro Nazionale Universitario di Calcolo Elettronico, Pisa, Italy, Computer Music Laboratory 11) Centre Nationale de la Recherche Scientifique - DR12, Marseilles, France, Laboratoire de Mecanique et d'Acoustique 12) Pompeu Fabre University, Barcelona, Spain, Institut Universitari de 1'Audiovisuel MOSART Project Objectives The goal of the MOSART network is to dessiminate young researchers within European research and postgraduate training activities, that address relevant problem areas within Sound and Music Computing, concerned with Machine Analysis and Understanding of Music Aspects of Sound, such as Timbre Space and Control and Virtualisation of Instruments. The Interactive Musical Performance situation will be addressed and Human Computer Interactive Conducting Tools will be studied. Furthermore the High and Low Level Representation aspects of music at both acoustic and at symbolic levels will be studied, in general and specifically for supporting conducting and composing tools. The project will in general study aspects of the use of computers in music analysis, digital music representation and computer assisted musical composition and performance. These efforts have as a main theme and goal Music Orchestration Systems, which will be addressed from both an acoustic analytic, from an interactive artistic and from an information technology and computer science point of view. Hence the acronym of the project, MOSART, Music Orchestration Systems in Algorithmic Research and Technology. Academia, industry and electronic art will work together on the issues, constantly baring identification of potential technology transfer to industry in mind. Identified partners within electronic industry and performing art will either be partly involved or be occasional observers of such technology transfer possibilities. An important sub-objective throughout the project will be observation of the educative aspects on the disciplines involved,

Page  00000003 attempting to define a standard, or minimal educative curriculum for teaching these subjects at a graduate level, including also applicability of each area of research to other non graduate pedagogical environments. MOSART Research tasks and research method A number of research tasks has been identified within MOSART, which are briefly described below. As co-ordinating node we are generally involved in every task, and in particular in the topics, described in the following section. Concerning Description of State of the Art. It should contain elements of the expertise of all groups in the consortium, and their counterparts beyond Europe. A major survey paper or a booklet will be compiled, where each chapter, aiming at the state of art within each group speciality and each task in the project, to be explained below. The resulting document is deemed to be a major reference in this field, and a most useful tutorial in both education and research. The survey will also serve to identify suitable educational curricula for teaching the subjects. Concerning Musical Sound Understanding, Timbre Modelling, Control and Virtualisation of Instruments. The method will take perceptual evidence and experimentation into consideration, and look for alternative descriptors of sound quality, than found in traditional acoustic analysis. The analysis by synthesis approach will thus be a prevailing technique in these tasks. Concerning the broader scope of modelling of phrases and of complete instruments, at least two approaches seems attractive: (a) the sheer physical based modelling, where metric and material constraints in the instrument, and in the sound transmitting environment, and in the physical ear, are taken into account; (b) using genetic algorithms and other heuristics in a 'virtual life' approach towards experimentation, psycho physics evaluation and further model building and consolidation. Both methods should be tried and compared. Concerning Detection of Human Motion and Interactive Musical Performance. The method will contain three major issues: (a) How to detect information in human body motion, such as dance and conducting, (b) how to link this detection to control mechanisms of musical control structures and (c) how to single out musical events in such control structures. Detection of human motion is itself a complicated task, where 'motion capture devices' is one methodology, and the search for vision based methods is another among several possibilities. The method will be concentrated in a breakthrough on the latter possibilities, while comparing and incorporating these in the former and more traditional motion capture paradigms. The method will take elements from motion sequence analysis and from biomechanics as well, in order to build a first prototype of a body motion detector, capable of revealing time evolving attitudes in human motion. The link and control structures between motion detection and the end goal, performance of music, will be studied in two ways: Taking the natural approach, that certain motions should be given classical meaning in conformance with conducting and dancing traditions, directly or context dependent; or taking the more innovative attitude, that chosen motions may deliberately be linked to certain and new meanings, and given control over musical events. These events may then be statically defined, or they may evolve over time, using a genetic algorithmic behaviour of control structures. Concerning Symbolic Recognition of Musical Patterns and Recomposition. The method will have to take into consideration perceptual existing evidence, and constantly pursue psycho physics evaluation of the performance of musical pattern detecting algorithms. For some reason it already appears, that some pattern recognition methods possess 'music analytic and expressive power', let it be some genetic methods, let it be 'brown noise', let it be other specific findings in the field. The method will involve comprehensive attempts to establish, why this is so, and what musical constructions are the basis for building good musical pattern recognisers. Concerning Computer Music Composition Tools. The study of data bases, and methods for organising musical structures and entities, will be obvious parts of the methodology. Data base organisation is meant to assist queries and rearrangements of certain data, and furthermore assist generation of new data with specified properties and in specified amounts. This is such a very basic property of both data bases and of composition, that a thorough study of 'musical databases' seems a must for computer assisted composition, of classical music, contemporary music and computer music. Concerning Technology transfer and Industrial exploitation. Several obvious initiatives may be taken: (a) Suggesting further market segments for the tools, already at hand; (b) Suggesting new products within the market segments, already established; (c) suggesting new company efforts, or new patents, on which new industry may be based. The task will not involve all of the economic and market analytic efforts, needed to start a new production, but be a scientists indication for possible technology transfer towards industry. 4 Courses A co-operation between the following institutions has been established in order to cover several aspects of Music Informatics, Music Acoustics and Computer Music. The aim is to allow students to float freely among the disciplines and gain insight from either angle. Likewise at research level, mutual inspiration may be gained. The co-operation is becoming formal at educational levels, in the sense that merits in Music Informatics may be transferred to curricula at the other institutions. In addition, students from below standing institutions regularly follow courses given by the music infromatics laboratory. The institutions are: 1) Department of Datalogy, University of Copenhagen 2) Musicology, University of Copenhagen 3) Computer Science Department, University at Arhus 4) Technical University of Denmark at Lundtofte 5) IT University of Copenhagen 6) The Design School in Copenhagen 7) Carl Nielsen Conservatory in Odense

Page  00000004 The music informatics group currently gives courses in Musical Signal Processing, Real-time Synthesis and Effects, Introduction to Music Informatics and Audio-Visual Perception. In recent years, the laboratory has supervised a large number of Ph.D., Post Doc and master thesis students. Typical topics have been analysis/synthesis and modeling of musical sound, watermarking, feedback elimination, symbolic pattern recognition of musical styles, generation of chords and variations, automatic performance of written music, adding feeling cues, interactive performing systems for real-time re-composition. Several of these activities are undertaken by students coming with an interdisciplinary background from, for instance, musicology, design and engineering. 5 Research Several long-term research projects are currently active in both music signal processing and symbolic notation aspects, but also more artistic aspects of music informatics are undertaken. In the signal processing side work are currently undertaken on improved sound analysis, monophonic and polyphonic segmentation and musical sound models. Furthermore, research regarding the real-time manipulations of sounds is performed. Touching the perception aspects, work is done on the importance of phase (Andersen and Jensen 2001), and HCI related research is undertaken to relate the signal and the perception of the signal to the model. In particular, the perception and discrimination of the parameters of the model presented in (Jensen 1999a) is studied. This research is used, for instance, in the classification of musical sounds (Jensen and Arnspang 1999). On a more artistic side, gesture caption, using image-processing techniques, virtual control and recomposition are studied. Regarding symbolic music notation research, work on extracting and classification of musical features from melodic material is performed. Another project under way, in line with several of the above-mentioned tasks, is research into developing a system to aid composition and real-time performance, based on gesture capture. In tandem, we have a Virtual Reality project up and running in collaboration with two other institutions, using video-based gesture capture both as a means of control and as a means of expression. Other applications of gesture to computer music are being investigated in collaboration with our partners in CNUCE (Pisa), DIST (Genoa) and DAIMI (Arhus) under the framework of the MOSART network. 6 Equipment On the audio side the Laboratory is equipped with high end studio equipment such as Brtiel and Kjaer microphones, Quad Electrostatic speakers, Studer DAT, eight track ADAT and several protools, running on Macs. At the MIDI level more customary equipment such as Roland, Ensonic and Macs are found. In case anechoic chambers are needed we reach major such installations at the co-operating Technical University of Denmark within one quarter by public transportation. On the video and other sensor part, we are in close collaboration with the computer vision laboratories at our institute and are also currently building new Virtual Reality facilities at our department, including a high-end surround sound installation. The Music Informatics studio is used for many purposes, including recordings, perception experiments, listening sessions and sound and music manipulations. The equipment is further used by the local university cabaret and many improvisation and informal music sessions. 7 Industrial and Cultural Relations Concerning industrial relations we have several partners in Denmark, among these TC-Electronics and Amazing Music World, with whom we have close collaboration and ongoing projects. These are found within timbre manipulation, score scanning, score handling tools and audio games. Projects may be short term feasibility studies as an preamble for major and joint fund raising such as MOSART, or it may be long term research in terms of funding for post Doc's. Development of end users product within the edutainment industry is being negotiated, but cannot be revealed at present. Within science of music, conservatories, design schools and performing art ensembles we have a likewise rapidly growing network in Denmark. Several courses, exhibitions and stage performing pieces are currently being set up, who need support from our efforts in Music Informatics. One example is an exhibition at Louisiana, a cooperation with the design school of Copenhagen: another is a composition based on a conic shape mirroring technique; another forthcoming example is an opera for the 200 year anniversary of Hans Christian Andersen, to be performed in Odense in the coming year(s). References Mosart 2000, http://www.diku.dk/research-groups/musinf/mosart/ Lindeberg, T., 1996, Edge detection and ridge detection with automatic scale selection, CVAP Report, KTH, Stockholm. Jensen, K., 1999a, Envelope Model of Isolated Musical Sounds, Proceedings of the DAFX, Trondheim, Norway. Johansen, P., 1994, On the classification of toppoints in scale space. J. Math. Imaging and Vision. 4, 57-67. Andersen, T. H., and K. Jensen, 2001, On the importance of phase information in additive analysis/synthesis of binaural sounds, Proceedings of the ICMC, Havana, Cuba. Jensen, K., 1999b, Timbre Models of Musical Sounds, PhD. Dissertation, DIKU Report 99/7, 1999. Jensen, K., J, Arnspang, 1999, Binary Tree Classification of Musical Instruments, Proceedings of the ICMC, Beijing, China.