Page  00000001 MMT TRAVEL KIT Dr. Rosemary Mountain Concordia University, Music Department 7141 Sherbrooke St. West, RF 312, Montreal, QC, Canada H4B 1R6 ABSTRACT The Multimedia Thesaurus is a playful tool for exploring sound and its latent and potential correlations with space, light, colour, image and movement. Its design is aimed to stimulate critical discourse about sound and multimedia, and to facilitate discussion between potential collaborators who may not share the same terminology, background experience or conceptual framework for describing sounds and music. Participants explore collections of very short sound and image files through the manipulation of physical objects linked to the files through barcodes. The sounds and images are sorted according to their perceived salient characteristics, whether relating to mood, association, or parameters of the audio signal itself. Each has an associated database entry in the computer with various types of information, updated at each "sorting". Although the main format of the tool is a room-size installation, the MMT Travel Kit is a portable prototype, where the objects can be sorted into trays and/or on trees by colour, keywords, or diagrams. For this demonstration, most of the clips will be drawn from contemporary electroacoustic and computer music compositions, including the FreeSound collection [4] and works by composers associated with the project. 1. INTRODUCTION The MMT Travel Kit is the portable version of The Multimedia Thesaurus, a project initiated in 2003 with start-up funding from Hexagram, a new Media Arts Institute in Montreal. The Multimedia Thesaurus is being designed to encourage an exploration of sounds and our perception of them, especially as they might be used in multimedia contexts. It presents users with a framework, recommended strategies, and specific banks of sounds and images to be "sorted" according to their salient characteristics.' A principal objective of the project is to increase users' sensitivity to and understanding of the different characteristics of sounds, their associations, and the effect of accompanying colour, shape, and movement. Long-term goals include the refinement of vocabulary for critical discourse of sound in multimedia and the development of a nucleus of researchers with enhanced awareness of sound and its potential in communicating specific ideas, moods, gestures, and images. In order to maximize its For an overview of the project in its initial conception, see [6]. effectiveness, the Multimedia Thesaurus is now being expanded into a coherent environment referred to as the Interactive Multimedia Playroom. By establishing a framework which is both playful and documentable, the Multimedia Thesaurus and Playroom aim to create a rich environment for reflection on the suitability of existing and proposed vocabulary and analytical tools and strategies for sound and music. The project is deliberately open-ended and does not assume the existence of an "average" or "ideal" set of responses to any given musical fragment or their association with particular images. On the other hand, it is expected to provide some hints about the degree to which certain reactions may be cultural or universal. The Multimedia Thesaurus benefits from having experts on its team drawn from various relevant fields of research such as musical imagery, rhythmic analysis, emotion, perception & cognition, and film music. It is also being designed to focus attention on other relevant research internationally, through a variety of means ranging from the establishment of links to relevant databases to the incorporation of particular software and hardware such as automatically-produced graphic scores and sensor-based instruments for manipulating sounds. 2. BACKGROUND / MOTIVATION The Multimedia Thesaurus developed out of a perceived need for analytical tools for music that is not adequately viewed through traditional notation. This includes improvised music of all sorts, as well as electroacoustic / computer music and sound design for media. It is also designed to focus attention on music whose attractions are less adequately revealed through traditional notation or analysis, for reasons of rhythmic complexities, timbral richness, microtonal inflections, etc. Most of us who have worked in collaborative settings or who enjoy discussions with friends and colleagues in different disciplines have been confronted with the inadequacy of words to express our reactions to different music and sonic effects. Traditional academic music courses have tended to encourage description through very specific terminology, whether dominant sevenths and deceptive cadences or comb filters and upper partials, and shy away from what are considered more subjective comments about mood and associations. In addition, the complexities of rhythmic form and our perception of temporal qualities are difficult to express given the apparently ephemeral nature of time and the number of conflicting metaphors used to discuss it. The problems are compounded in the areas such as

Page  00000002 electroacoustics, improvised, and traditional nonWestern musics which do not rely on written notation at some point in the chain between concept and reception. In such cases, any discussion has to acknowledge that two people's memory of the same passage is dependent on the perception of the organizational framework and salient characteristics, as these factors influence the way in which the musical information is classified in the listener's mind for later reference. Such perception is in turn highly dependent on the listener's education, experience, and personal aesthetics. It is not surprising, therefore, that an examination of how people hear different passages has been neglected due to a lack of appropriate means of investigating the subject and an assumption that the bewildering complexities of the field are overwhelming. However, given the increasing use of music and sound in interdisciplinary contexts - whether web design, video, dance, or installation art - the need to be able to discuss such issues is becoming more pressing. Students who have no particular background in music are trying to integrate music into their work without guidance; those of us with expertise in the area can often hear that the results are weak, but are frequently tongue-tied when it comes to articulating the reasons for that weakness, and may in the end say nothing at all, to the detriment of development in the artist and in the field itself. The current project is motivated by the conviction that any attention to these issues will have a positive impact on our understanding, teaching, and creation of musical and multimedia works. 3. DESCRIPTION 3.1 Overview Just as the traditional word thesaurus does not define words, but rather groups them by association, the Multimedia Thesaurus does not aim to define sound characteristics, but rather to explore the ways in which individuals and communities may interpret and describe sounds with reference to other sounds, to images, colours, movements, space, moods, and atmospheres. Unlike the traditional thesaurus of words, the Multimedia Thesaurus allows users to build their own groupings and lists of associations, as well as drawing on the suggestions of previous and concurrent users. In the Travel Kit version of the Multimedia Thesaurus, around a hundred hand-held objects, each with a unique bar code, represent an array of sound and visual "clips" of around ten seconds' length each. Scanning the barcode on the clip with a lightweight wireless Bluetooth barcode scanner displays a still or moving image on a screen or triggers playback of sounds from speakers or headphones. Participants (often in groups of two or three) choose a subset of sounds and images with which they wish to play, and then "sort" them, according to their characteristics. Like a game, this sorting can proceed in a variety of ways. Each clip has an associated database entry in the computer, listing various types of information relating to it - from source and copyright information to objective and subjective descriptions of its characteristics. We are currently working to develop an efficient system which will permit the gathering of information from each time a person or group sorts a set of clips, allowing for the accumulation of rich data banks. Eventually, we should be able to have a substantial set of data for psychologists and others to peruse. In the interim, people will be able to test specific reactions and classification tendencies for small subsets of the clips. Although similar kinds of testing are common in music psychology, we suspect that the playful context and the weighting of participant profiles to composers and other creative artists will encourage the participation of "subjects" who are less likely to volunteer for laboratory tests. In addition, we suspect that the physical aspects of the "sorting" process and the allowing of individual participants to contribute their own labelling criteria may in the long run provide a more accurate reflection of the diversity of associations and reactions to sounds and sound/image correlations. 3.2 Travel Kit version The Travel Kit version, which focuses on sorting the clips into trays or bins, is considered the basic version of the game. In order to contextualize this version within the larger concept of the Interactive Multimedia Playroom, the demonstration is complemented by notated images of the (multiple-) room-sized installation, where clips can be arranged on links of a chain into a large 3-dimensional grid, manipulated according to certain basic parameters, and appreciated in a variety of settings. The labels - whether for the portable or larger installation version - may be selected from recommended lists or invented by the users, and may consist of colour swatches, diagrams, words, etc. Even without determining labels, the activity of evaluating the degree of similarity between any two or more clips can be monitored, due to their being placed into the same or adjacent bins. An additional format of the MMT Travel Kit is being designed to incorporate "trees" which enable the clips to be grouped according to similarities at three or more hierarchical levels: they can be placed close together on the same branch, or on different branches of the same tree, or on different (closer or farther-away) trees. 3.3 Clip selection and categories The clips gathered for the ICMC-Barcelona demonstration aim to represent a variety of sounds and sonic collections that can be found under the umbrella of electroacoustic and computer music. They include fragments drawn from different aesthetics as well as providing a range of types of sounds: typical MIDI keyboard presets, granulated sounds, musique concrete drawn from urban and natural sounds, treated and untreated vocal sounds, etc. Although many of the sounds will be taken from compositions, some will be

Page  00000003 drawn from the categories of signs, signals and sound effects. In addition, the array will contain a range of recording qualities, and this will be further explored through a variety of playback systems of different degrees of quality / accuracy to the original signal. The clips are sorted by the participant according to pre-determined categories, which may follow standard psychological divisions such as active/passive activity level, positive/negative valence level, etc. Other more readily accessible sorting parameters could include scales such as "familiar / strange", "urban / rural", or groups such as "patterned" "abstract", etc. A more intuitive-minded participant may prefer to sort by association with colour or mood, while musically-trained users may be comfortable with sorting by rhythm / form, tonality, and genre. For the mobile MMT Travel Kit, each tray or bin may be assigned labels by word, diagram, or colour, to facilitate the sorting. Descriptive categories already in use by various music information retrieval systems, as well as those used traditionally by psychologists, are being incorporated into the database categories, and the research is being linked to related projects internationally (such as the deMontfort-based EARS project [2] and Poissant's expanding Dictionnaire [1] des Arts Mddiatiques). The project is being designed to maximize its compatibility to other research, so that those involved in MIR studies and music psychology, for example, could eventually use the Thesaurus as a "testing ground" for terminology, software, etc. 3.4 Objectives Once any given "sorting" is completed, the results will be added to a data pool for future analysis. However, it is already evident that when the sorting is undertaken as collective activity of two or more people, the process of articulating one's own opinions to the others and seeing the differences between the various users' categorizations can yield immediate and rich information. The usefulness of this information is very high when the different people in question are contemplating a collaboration, because in the process of communicating with each other, a good deal can be learned about the others' aesthetic preferences, perceptions, etc. The Thesaurus can also be used by professor / researchers as a tool for focussing attention on particular categories or terms in need of more reflection and better definition, such as musical gesture and texture. In fact, the initial collection of clips was initiated through the search for examples to illustrate various aspects of music and multimedia analysis, and student input at various stages has been invaluable. It is clear that by confining the duration of any given clip to around 10", many aspects of formal structure and other time-dependent elements may be overlooked. However, as a first step, it seems that much information can be gathered even from these artificial constraints. In particular, the aptness of any particular sound / image correlation can often be determined in a very short time period, and until this is understood, the longer-duration matches may make appraisal difficult. In the Playroom version, the Thesaurus component is being complemented by various other parts including a library area where people can browse CDs, books, journals, and websites. 3.5 Team The team involved in the Multimedia Thesaurus and Interactive Multimedia Playroom is growing continuously and significantly. Some team members are more involved with the design of the physical and virtual aspects of the project, some contribute expert information from their own fields of specialization as consultants, some are exploring the ways in which their own projects can be integrated into the Thesaurus in order to encourage a cross-fertilization, while some are simply eager to be "beta-testers". At the moment, consultants include internationally-renowned specialists from various areas including music psychology (Dr. John Sloboda), auditory perception (Dr. Al Bregman and Dr. Stephen McAdams), musical imagery (Dr. Rolf Inge Godoy), film music psychology (Dr. Annabel Cohen), analysis / musicology of electroacoustics (Dr. Leigh Landy and Marcelle Deschenes), gesture (Dr. Marcelo Wanderley) and computer science-VR (Dr. Sudhir Mudur). Colleagues from Hexagram [5] and Concordia University are also collaborating from the perspectives of their various disciplines which include film, animation, music, dance, virtual communities, and computer science. Research assistants also represent a number of disciplines, and are helping in all facets of the proj ects. 3.6. Future directions Although part of the project's design is based on the premise that the physical characteristics of the sorting grids and boxes and accompanying paraphernalia can provide a much richer environment than that of a typical experimental psychology lab or the virtual sketch of a 3 -D space on a computer screen, the project is intended to be developed in ways that can be easily transferred to other such formats. The database is being designed to facilitate analysis of people's opinions on the salient characteristics of individual clips and their perceived similarities to others; in addition, previous studies with musical examples will be able to be transferred to the format of the Thesaurus (Travel Kit or Playroom version) to permit re-testing in a richer environment. In addition, investigation is already taking place on the possibilities of creating virtual models. These might range from the presentation of the clips on a webspace for virtual sorting, to the creation of a VR or mixed reality space. The various coexisting formats of the project will then allow an examination of the effect of virtuality on the sorting process.

Page  00000004 Although the Interactive Multimedia Playroom will exist in the first instance in only one space, the project should be able to be reconstructed [and reconfigured as necessary] in other places. The MMT Travel Kit will then be seen as one of several manifestations of the project. It is anticipated that it will become in some ways the "Trial Version" to entice potential clients to subscribe to the full installation package. 4. CONCLUSIONS The demonstration is being designed not only to explain the concepts behind the project but also to engage ICMC participants in the sorting process and its accompanying reflection and discourse. Participants are encouraged to reflect and comment on the appropriateness of the terminology in specific and global instances, so that the project's findings can begin to be fed back into such research. The demonstration is also meant to draw attention to and stimulate interest in the fuller installation / environment version currently being developed in Montreal (with public inauguration in October 2005 prior to the EMS-05 conference [3]) and to encourage those working in intersecting areas of research to imagine ways in which their own research can be linked to the Interactive Multimedia Playroom. Because its format encourages evaluation not only of sounds, images, and their correlations but also the context and the hardware being used, the Thesaurus is potentially an excellent place for introducing different perspectives on issues relating to perception of sound and sound/image combinations. In addition, composers or analysts who are curious about the variety of reactions to specific segments of a given piece may discover that the Thesaurus provides an easy means of gathering data on such subjects. Although the project is not designed exclusively for the examination of computer music and electroacoustics, it is meant to address issues that are relevant to those fields by placing emphasis on the importance of the aural signal (rather than on written notation) and allowing for a focus on different aspects of structure (narrative, non-linear, traditional forms, etc.), mood, atmosphere, and associations, as well as texture, gesture, timbre, and other aspects of the sound signal which are often overlooked in acoustic music. Because the Thesaurus encourages a clip selection according to different parameters regardless of whether the clip is drawn from acoustic music, electroacoustics, or nonmusical sounds (ranging from electronic signals to recordings of natural sounds) it is consistent with the arguments of the principal investigator [7] against the "ghettoization" of the electroacoustic / computer music field. REFERENCES: [1] Dictionnaire des Arts Mddiatiques site: [2] EARS site: [3] EMS-05 site: [4] FreeSound site: [5] Hexagram site: [6] Mountain, R. "Flexible Frameworks: The Multimedia Thesaurus". V Triennial ESCOM conference (European Society for Cognitive Sciences of Music), Honnover, Germany, Sept. 2003. [7] Mountain, R. "Theories Market: Open for Trading", OrganisedSound, 9/1: 15-26.