Page  00000423 NOISESCAPE: AN INTERACTIVE 3D AUDIOVISUAL MULTI-USER COMPOSITION ENVIRONMENT Dr Mick Grierson Goldsmiths College, University of London Department of Music ABSTRACT This paper presents a prototype 3D first-person composition and improvisation system for Windows and Macintosh computers. Designed to look and behave like contemporary 3D first-person computer games, the Noisescape system places users in an abstract virtual world complete with physical modelling, collision detection, and 3 dimensional movement. There are no predetermined constraints on the environment, object properties or interactions. Users can create, destroy and combine elements with varying physical attributes in order to structure complex music systems. Both the objects and the environment can respond to user behaviour in real time using simple adaptive algorithms. In addition, Noisescape demonstrates potential for use in systems programming. This work should be of interest to those in the fields of electronic and computer music, audiovisual composition, networked performance, and multiplayer games design. This research is part of the AHRC funded project, "Cognitive and Structural Approaches to Contemporary Computer Aided Audiovisual Composition". 1.INTRODUCTION Noisescape is a prototype 3D first person multiplayer composition and performance environment created in Max/MSP/Jitter through the application of physical modelling, games design and audiovisual composition techniques. Musical events are represented by interactive 3D physically modelled objects in a virtual space. Connections between these objects allow for the building of algorithmic computer music systems which can be stored and re-loaded by performers in the virtual space. 2.PREVIOUS WORK Richard Bartle, (MUD, Bartle, R., 1978) has written extensively on the history and theory of multiplayer games [2], and also on the creation of multi-user virtual environments [3]. Maze war (Colley, S., 1972-73') is regarded as the first 3D networked computer game. Contemporary environments are increasingly being used by artists as a creative resource (see Clarke, A and Mitchell, G., 2007 [10]) 1 http://en.wikipedia.org/wiki/MazeWar accessed 04/05/07 Multi-user environments are highly inclusive, becoming popular with the general public. On-line virtual spaces such as Second Life are the subject of current research in archive management [11] and media search2 (as part of the MUSCLE research group). Second Life's user base currently stands at approximately 6 million accounts. Some significant research has been carried out in the area of virtual music environments, specifically for acoustic visualisation (Wenzel, E.M., Stone, P.K., Fisher, S.S., Foster, S.H., 1990 [17]) and interactive sound processing (Bargar, R., Choi, I., Das, S., Goudeseune, C., 1994 [1] (using the CAVE3 virtual reality environment)). More recently, research into physical modelling synthesis via gesture control (Howard, D.M., and Rimell, S., 2003 [12]), and adaptive music systems in visual simulation environments (Spector, L. & Klein, J., 2002 [16]) have generated renewed interest and fresh perspective on initial work. 3.THEORY AND CONTEXT 3.1.Collaborative 3D Interfaces In normal circumstances, computer environments are designed to be operated by a single user. In addition, single-user software is often conceptual in approach, as opposed to experiential. An alternative approach can be found in immersive 3D on-line multi-player game environments. In these cases, control interfaces are mainly based on paradigms of movement in space. Research highlights the value of developing 3D interfaces based on real-world interactions [5]. In addition, experiential interactivity is a main focus of collaborative interface design. There are still problems with this approach [6]. The required combination of keyboard and mouse control requires some skill to master. Also, it can cause motion sickness in some users. Despite this, large numbers of people are already familiar with these methods of display, control and collaboration. Noisescape has developed from the notion that audio, visual and audiovisual composition could be carried out in an interactive virtual space, thereby providing electronic and computer music with a consistent environment, similar to that which exists in the real world for acoustic composition and performance. Most importantly, 2 http://ispaces.ec3.at/muscle.php accessed 04/05/73 3 http://en.wikipedia.org/wiki/CaveAutomatic_Virtual_Environment accessed 04/05/07 423

Page  00000424 however, Noisescape is not a virtual world which mirrors our own. It instead aims to be consistent with the extreme possibilities made available through electronic and computer composition approaches - an effective audiovisual electronic music universe. 3.2. Collaborative Electronic Music Performance Networked performances often depend on users developing their own systems, deciding on a specific set of mappings, and then sending the required data to others (examples of this approach include Lemma 1 (Sorensen V., Steiger R., and Puckette M., Thessaloniki, Sep. 27, 1997) [13])). Attempting to transmit all information about every system to every user requires a high level of standardisation. To achieve this, Noisescape places each participant in the same virtual environment. This allows for a high degree of collaboration. Pieces can be constructed both collaboratively and by individuals for reliable and repeatable performance. All elements within the system can be arranged so that they produce repeatable sequences of material. Each state or moment (arrangement) can be stored and recalled at any time by users operating within the world. In this way, pieces (or systems) can be scheduled (or programmed) in any fashion the users wish. 3.3.Audiovisual Composition In conventional electronic music performance, works are diffused in real space. However, with increased levels of electronic treatment; the relationship between cause and effect can become harder to detect (it is acknowledged that this is not significant with respect to acousmatic music). Audiovisual composition approaches go some way to solving these issues by treating all sonic and visual events as a single perceptual unit. In addition, physical gestural interfaces can also play a part in this process (although this approach may be less successful due to the lack of reciprocal visual feedback from audio generators). By placing the spectator within a 3D electronic environment, each sonic element and algorithm can be (in theory) represented by an object in a chain of events. Complex sonic interactions can be directly perceived audiovisually. 3.4.Physical Modelling Noisescape utilises physical modelling techniques for almost all interactions, the exceptions being participant propulsion and orientation, which are only partially modelled. This is consistent with contemporary approaches to the constnuction of first person environments [6]. One significant advantage of this dualphysics approach is that it allows for the construction of systems based pieces which rely on real-world style interactions, whilst also allowing the user to move freely. The implications of this type of collaborative audiovisual immersion is of particular interest with respect to the perception of causation and working memory, which is the subject of current research (see Representing Causation, Wolff, P., 2007 [18]). 4.METHOD 4.1.Development Tools The primary development environment for the Noisescape project is Cycling '74's Max/MSP/Jitter, based on Miller Puckette's Patcher environment (Puckette, M., 1988) [14][15]. This means that the system can be prototyped quickly and flexibly across several platforms. At present, the Noisescape system utilises Nicolas Montgermont's open source MSD' (mass spring damper) externals for implementing physical modelling and collision detection. These are optimised versions of the PMPD externals by Cyrille Henry2. MSD was chosen for it's cross format compatibility. 4.1.Creating a First Person Viewpoint in Max/MSP/Jitter The user's viewpoint is calculated by taking point A as the camera position, and a second point B as the 'lookat' position. These points are spaced one unit apart along the z axis. Orientation is calculated by rotating the lookat position around the camera position using a matrix rotation. A unit vector is then calculated between point A and point B, and this value is multiplied by the current velocity. By calculating the unit vector at a rotation of 90 degrees from the lookat position, sideways movement is possible. In addition, the lookat position can be above or below the camera position, allowing the user to fly up or down. Control of the perspective viewpoint has been designed to be consistent with contemporary first person computer game environments. The lookat position is controlled using the mouse. The position is altered by pressing the forward, backward, left or right keys (w,s,a,d respectively). Pressing any of these keys will cause the user to accelerate to maximum velocity in the appropriate direction, and two keys can be pressed at once in order to travel diagonally. By pressing the shift key, the user can increase the maximum velocity - an effective 'running' mode. When approaching maximum velocity, there is an intentional increase in lag whilst orienting the lookat position due to a gain in momentum. 4.2.Interacting with Physically Modelled Sound Entities At present, interaction follows the simplest model possible whilst maintaining hybrid grabbing and manipulation functions (see Bowman, D. and Hodges, L., 1997 [7]). Modifier keys can then be used to alter the properties of entities. Entities are represented as spheres by default. Each entity is linked to a specific sound by referencing a wavetable buffer. Users can change the original pitch by holding down the p key and moving the sphere up or down. Users can also select a loop mode (o key) and enable a re-trigger function (r key). In addition, users can enter scratch mode by holding down the c key and moving the sphere left or right. This allows users to scan through the file in order to set the start point. By default, all objects have a standard amplitude envelope with a 1 ms attack and 1000 ms release. These values can be changed 1http://www.basseslumieres.org/projets.php?id=15 accessed 04/05/07 2 http://drpichon.free.fr/pmpd/ accessed 04/05/07 424

Page  00000425 by altering the shape of the sphere. Holding down the e key whilst moving the sphere left or right will increase the width of the object, making the sphere into an ellipsoid, thereby increasing the attack and release in ratio. Moving the sphere up and down in this mode will increase the maximum potential amplitude. The lowpass filter's cut-off frequency can by set by holding down the f key and moving up and down. A basic filter envelope can be created and set by moving left or right, with parameters identical to the amplitude envelope. When an entity is struck against another entity (including stationary entities), it sounds with a volume equal to the force it was struck with (collision forces are calculated by the MSD library). At the most basic level, users can drop, throw and hit entities, or cause entities to strike each other. Entities can be placed in specific positions, and then triggered in sequences, for example in cascades, or through the use of a trigger entity - a playback head. 4.3.Adapting Entities and Environments Entities can be placed in 'adapt' mode by pressing t. If the entity then bounces against another object, it will become more springy. If it is left alone, it will become less springy. This will continue until it reaches its maximum or minimum value, or adapt mode is turned off. In this way, entities can retain their energy. This allows for the creation of repeating sound textures, such as rhythms and loops. In addition, if a variety of entities are placed in adapt mode, they will continually react to each other, taking on different properties depending on the state of the other entities in the system. This can create large artificial adaptive music systems which have interesting properties. In addition, Entities which have been trained to exhibit no motion whatsoever can be used as building blocks for the environment, such as walls, ceilings, channels, mazes, gates and other stationary elements, whilst retaining their sound making properties. This allows users the autonomy to create large flexible environments intuitively. 4.4.Creating Sound Systems Through Links Entities can be linked together to form 'sound systems'. There are two main types of sound system links. Physical links, where objects are bound together by a material link with properties of its own, and sonic links, where the movement of one object affects the sonic properties of another. Physical links are generated and controlled through the MSD library linking system. Links are created using the link command, which is triggered by holding an entity near to another entity and pressing the 1 key. This creates a rigid link between two objects. Grabbing and moving one of the entities will increase or decrease the initial length and elasticity of the link. If it is pulled quickly, the link becomes more elastic. If it is moved slowly, it relaxes, eventually becoming a fixed length. When the entity is dropped, the link is fixed relative to the other links connected to the object, and can still be changed through the use of the 'adapt' function. Once an entity is linked to another, it cannot be unlinked. However, any entity can be destroyed at any time by grabbing it and pressing the right mouse button. Pressing this button when no entity is nearby creates a new entity. Sonic links are connections between the sonic properties of entities. The movement of any entity can be used to control any sonic property of any entity, including its own. In this way, the motion of one entity can be used to vary the pitch, filter, start position or amplitude of another. Using sonic links, it is theoretically possible to use entities as elements in a large physically modelled virtual reality modular synthesiser, with each entity as a controller. Sonic links are created in a similar way to physical links. The user can grab an entity, hold it close to another entity, and press x to create the link. If no object is nearby, the entity attempts to create a sonic link with itself. The following keypress defines which sonic attribute the motion should be mapped to. At present, movement can be applied in one dimension to either the amplitude, the pitch, the filter, or the loop start position (by pressing a,p,f or c respectively). Through careful combinations of sonic links, 3D virtual sculptures employing frequency modulation, amplitude modulation and granular synthesis can be created. In this way, the Noisescape model can be used as a very simple embodied 3 dimensional audiovisual version of Miller Puckette's patcher-based programming paradigm. 4.5.Spatialisation There are two ways in which audio can be spatialised in Noisescape. First, sound can be heard relative to the user's position in 4 channel surround (which is down-mixed to 2 channels automatically). This means that each participant hears all audio within a fixed radius attenuated by 1 (maximum) minus the absolute distance from 0. (the users position), with each entities position along the relative x and z axis used to define the amplitude value in each speaker, x+z=0 being equivalent to a value of 1 (maximum). As the sound objects can be seen, visual cues aid in the perceived positioning of each object (see Chion, audiovisual Magnetisation, Chion, M., 1995 [9]). The second method of sound spatialisation is a single perspective mix. This is useful if a composer or performer wants to create a particular experience that will sound identical for each participant. This method is also useful for certain types of music, where specific events need to be heard in a particular way in order for a piece to be successfully communicated. Individual entities and sound systems can then be placed in ideal locations so that the piece unfolds in the intended fashion. 4.6. Client-Server Model Noisescape utilises a client server model in order to implement a multi user environment (at present without latency compensation [4], although there are plans to explore this approach in some networked contexts - for discussion see Bemier, Y.W., 2001). It does this using the built in udpsend and udpreceive networking objects bundled with Max/MSP/Jitter. A central server program keeps track of every element in the system including all 425

Page  00000426 users, entities and environmental conditions. This also includes all physical attributes and links. In practice this means that the MSD library only runs on the server, and none of the clients need to have the library loaded. It also means that the server itself can dedicate all its processing power to tracking physically modelled objects, links, forces and media attributes without the need to render graphics, download images or process audio. All changes to the environment that are handled by MSD are immediately transferred to all other clients in the system. 5.FUTURE WORK The application of collaborative 3D environments to the development of an object-based programming paradigm is of great interest. Benefits include enhanced visualisation of programming for teaching, learning, composition, performance and research. Most importantly, Noisescape has been developed in order to facilitate the exploration of cognitive and structural approaches to contemporary audiovisual composition. As such it provides a framework for the development of experimental designs and the analysis of user behaviour. The results of these experiments will be the subject of future publication. 6.REFERENCES [1] Bargar, R., Choi, I., Das, S., Goudeseune, C., "Model-based Interactive Sound for an Immersive Virtual Environment" Proceedings of the International Computer Music Conference, 1994. [2] Bartle, R., Interactive, Multi- User Computer Games. BT Martlesham Research Laboratories, 1990 [3] Bartle, R., Designing Virtual Worlds. Indianapolis: New Riders, 2003 [4] Bernier, Y.W., "Latency Compensation Methods in Client/Server In-game Protocol Design and Optimization", Proceedings of Game Developers Conference, 2001 [5] Bowman, D. A., Kruijff, E., LaViola, J. J. j., & Poupyrev, I. "An Introduction to 3D User Interface Design". Presence. Teleoperators and Virtual Environments, 10(1), 2001 [6] Bowman, D., Davis, B, Badre, A., &Hodges, L. "Maintaining Spatial Orientation during Travel in an Immersive Virtual Environment". Presence. Teleoperators and Virtual Environments, 8(6), 1999 [7] Bowman, D., & Hodges, L. "An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments". Proceedings of the ACM Symposium on Interactive 3d Graphics, 1997 [8] Brooks, B. M., "The Specificity of Memory Enhancement During Interaction with a Virtual Environment" Memory, Volume 7, Issue 1 January 1999 [9] Chion, M. (Trans. Gorbman, C. 1994), Audiovision. Sound on Screen, New York: Columbia University Press [10] Clarke, A. & Mitchell, G., Video Games and Art. Intellect Books, 2007 [11] Grassian, E., Trueman, R.B.,"Stumbling, Bumbling, Teleporting and Flying? Librarian Avatars in Second Life". Reference Services Review, Volume 35, Number 1, 2007 [12] Howard, D.M., and Rimell, S. "Gesture-tactile Controlled Physical Modelling Music Synthesis", Proceedings of the Stockholm Music Acoustics Conference, SMAC-03, Vol. 2, 2003. [13] Puckette, M. "Pure data: Recent Progress", Proceedings of the Third Intercollege Computer Music Festival, Tokyo, 1997 [14] Puckette, M., "Pure Data". Proceedings of the International Computer Music Conference, 1996 [15] Puckette, M., "The Patcher." Proceedings of the 1986 International Computer Music Conference. San Francisco: Computer Music Association, 1988 [16] Spector, L. & Klein, J., "Complex Adaptive Music Systems in the Breve Simulation Environment". In Bilotta et al. (eds), Workshop Proceedings of the 8th International Conference on the Simulation and Synthesis of Living Systems, 2002 [17] Wenzel, E.M., Stone, P.K., Fisher, S.S.,Foster, S.H., "A System for Three-dimensional Acoustic 'Visualizations' in a Virtual Environment Workstation". Visualization, 1990. [18] Wolff, P., "Representing Causation", Journal of Experimental Psychology: General, Vol. 136, No 1, 2007 426