Page  00000437 Musical Performance Over Internet2 Using The AccessGrid Charles Nichols', Scott Deal2, Timothy J. Rogers3, Jimmy Miklavcic4, Beth Miklavcic4, Many Ayromlou5, Robert Wachtel', Paul Mercer2, Joe Humbert3, Rob King5 'Department of Music and Information Technology Office, University of Montana, 2Arctic Region Supercomputing Center, University of Alaska Fairbanks, 3Envision Center for Data Perceptualization, Purdue University tjrogers, 4Center for High Performance Computing, University of Utah, 5Rogers Communications Centre, Ryerson University mayromlo, Abstract Artists and technicians at seven universities collaborated on an interactive multimedia dramatic work, that included acting, performance art, virtual reality, animation, motioncapture dancing, and electroacoustic music performance, streamed between six venues over Internet2. The music for this distributed performance included two structured improvisations between a percussionist in Alaska and an electric violinist in Montana, a piece in which the position data of a dancer in a motion-capture system in Indiana was used to trigger and select samples from soundfiles in Montana, and two solo pieces for percussion and tape and one for electric violin and interactive computer processing. 1 Introduction As a part of the Loose Minds in a Box Interplay, an interactive videoconferenced multimedia collaboration between artists and technicians at seven universities, three kinds of music were performed over Internet2, using the AccessGrid. For two of the scenes of the Interplay, electric violinist Charles Nichols, at the University of Montana, and percussionist Scott Deal, at the University of Alaska Fairbanks, performed structured improvisations together, while hearing each other over monitors, and seeing each other in projected video streams. For another scene, dancer Joe Hayes, at Purdue University, danced in a motion-capture suit, sending his position data as MIDI data, to MaxMSP programming at the University of Montana. His position data was used to generate granular synthesis of five soundfiles, which was mixed and processed in real-time by Nichols. For three other scenes, electroacoustic solo pieces were performed by Deal and Nichols, as accompaniment to the other dramatic elements of the work. The music for the Interplay was developed during videoconferenced rehearsals over several months, and presented in a series of performances, streamed between six venues, in April of 2005. Later, in August and November of 2005, Loose Minds in a Box was performed for multiple sessions at the SIGGRAPH and SCGlobal conferences. For each performance, live performers at each site contributed their part of the Interplay in front of a local audience, while video streams of the performers at the other sites were projected in the venue. At the same time, a mix of all of the audio and video streams was broadcast over the Internet. re 1. Jimmy Miklavcic mixing video streams of performers in Loose Minds in a Box 2 Background 2.1 AccessGrid The Access Grid~ is an ensemble of resources including interfaces to Grid middleware and to visualization environments, multimedia large-format displays, and presentation and interactive environments. These resources are used to support group-to-group interactions across the Grid. For example, the Access Grid is used for large-scale distributed meetings, collaborative work sessions, seminars, lectures, tutorials, and training. The Access Grid thus differs from desktop-to-desktop tools that focus on individual communication. 437

Page  00000438 The Access Grid (AG) is now used at over 500 institutions across 47 countries. Each institution has one or more AG nodes, or "designed spaces," that contain the highend audio and visual technology needed to provide a highquality compelling user experience. The nodes are also used as a research environment for the development of distributed data and visualization corridors as well as for the study of issues relating to collaborative work in distributed environments. The Access Grid technology was developed by the Futures Laboratory at Argonne National Laboratory and is deployed by the NCSA PACI Alliance. The Futures Lab continues to conduct research into ways to improve the Access Grid, for example, to increase the scalability and to enhance the user interfaces ( 2.2 ArtGrid ArtGrid is an informal consortium of creative and research institutions that utilize Access Grid technology for artistic purposes. This includes telematic performance, remote MIDI control, remote motion-capture control, distributed virtual environments and collaborative communications. University of Utah, University of Alaska at Fairbanks, University of Montana, Purdue University, University of Illinois at Chicago, University of Maryland, and Ryerson University are some of the institutions currently active in ArtGrid. 2.3 Interplay An InterPlay, developed by the Another Language Performing Arts Company (, is a multi-faceted telematic event that consists of two or more performances that occur simultaneously at multiple sites worldwide. The performances are concurrently captured, mixed, digitized, encoded and streamed onto the network. The director manipulates each video stream to appear in any of several video playback windows. This creates a work that takes individual events and weaves them into a multilayered distributed tapestry. Each artist has his/her thought process that leads to his/her artistic performance. The director moves that into another level by taking those performances and incorporating them into his own thought and creative process. InterPlays should be viewed as a painting in motion. A myriad of colors, text, shapes and textures float about the framed video space to the resonance of sounds, music and words. Within these visual and audible constructions, stories hover and pass through the viewer's thoughts, while in the overall fabric there is no preconceived story. Each satellite venue shares a portion of a common idea but is experienced differently depending on each venue's environment and technology. Images of the performers add the human dimension to the visual fabric, teasing the viewer into the possibility of a narrative, but stopping just short of telling an identifiable tale. 2.4 Loose Minds in a Box Within the ArtGrid community the InterPlays are a performance format created by Another Language Performing Arts Company. Development began in August 2004 for the third InterPlay: Loose Minds in a Box, with seventeen artists and twenty-six technicians coming together to create the work. After dealing with the subject of hallucinations in the previous InterPlay, director Jimmy Miklavcic was interested in exploring aspects of multiple personalities or schizophrenia. In order to make the performances more conceptually friendly to the general public, artistic director Beth Miklavcic came up with an abstraction of the idea, Loose Minds in a Box, allowing for a broad interpretation of this theme. Her hope was to have several different streams from around the country participate in exploring different personality restraints, freedoms or external perceptions. Loose Minds in a Box was divided into six scenes, each containing different acting, performance art, motion-capture dancing, computer animation, virtual reality, and musical performance, based on variations of the central box theme. 3 The Imprisonment of Thought and Let Loose the Mind Figure 2. Charles Nichols performing The Imprisonment of Thought as seen at the University of Montana For the second and fifth scenes of the Interplay, The Imprisonment of Thought and Let Loose the Mind, Charles Nichols and Scott Deal performed two separate, but motivically related structured improvisations, for processed electric violin and multiple percussion. Deal performed in the Arctic Region Supercomputing Center Discovery Lab, a virtual reality online CAVE environment, while Nichols performed in a videoconferencing room at the University of 438

Page  00000439 Montana. Both could see and hear each other, as well as the other performers. of multiple, networked, performance frames. The concept of "performance frames" was used by Meador et al. in order to divide an overall production into "discrete performance spaces, or 'frames', based on the specific properties inherent in each medium, i.e., stage, silhouette, video, virtual, feedback, and mocap." For the InterPlay project, the emphasis was primarily on extending the capabilities and influence of the "virtual frame" and "mocap frame" by exploring how mapped motion data could impact the performance, not only locally and virtually, but also remotely. A dancer at Purdue was fitted with a costume that included a nineteen-point marker set capable of feeding positional data to a real-time optical motion-capture system. Though data was captured from all nineteen markers by the system, only four markers were used to drive the performance elements in Loose Minds in a Box: the lower back, left wrist, right wrist, and front of the head. These markers established a rudimentary upper-body control structure, where the potential for gesture-based actions could be explored. figure 5. Scott Deal pertorming Let Loose the Mind as seen at the University of Alaska Fairbanks Although there was a varying amount of delay, ranging in the hundreds of milliseconds, between the two sites, the performers took on the challenge of performing rhythmically coordinated music at a fast tempo. Maintaining ensemble was sometimes challenging, but after several rehearsals, the performers were able to compensate for the audio delay, when it occurred, with visual cues and strict adherence to the initial tempo. This experience was similar to Nichols' performance with two other remote musicians, as a part of Chris Chafe's SoundWIRE project, but was aided by the addition of video streams. It was speculated that the musicians were able to play under the varying acoustic circumstances, because of their experience performing computer music with a variety of sound sources. This could be an interesting and fruitful area for quantitative research. 4 The Blue Box In the sixth scene, The Blue Box, data from a dancer in a motion-capture system at Purdue University was translated into MIDI data, and sent across Internet2 using the AGMIDI Access Grid service developed at Ryerson University, to a MaxMSP patch at the University of Montana, which used the data to trigger and select samples from five soundfiles, as a musical accompaniment to the other dramatic elements. 4.1 Motion Capture With the purpose of extending the framework for the "nesting" of performance frames in a live event (Meador et al. 2004), the motion-capture team at Purdue University's Envision Center joined the InterPlay project and utilized their Access Grid node to experiment with the presentation Sigure 4. joy nayes aanclng in a motion-capture suit as seen at Purdue University The motion-capture system used for the performance was comprised of six infrared cameras from Standard Deviation (, Motion Captor RT software from STT (, and Alias MoCap (, which was used for real-time display and manipulation of the data. The data streams being generated by the markers were parented and constrained to objects in the virtual scene, including nine cubes, particle systems, and 2D images of emitted text. The interactions between dancer and objects were based on simple vector relationships, distances, and the weighting of values driving parent/child connections. These relationships created a gestural-control model for the dancer to influence various properties of the virtual objects, primarily translation and rotation. 439

Page  00000440 The positional data of the markers were also monitored and filtered to create a spatial mapping structure, that sent a specific range of MIDI pitch and velocity values to Alias MoCap's real-time MIDI export device. The device streamed the MIDI data to the AGMIDI service which routed it to the University of Montana's MaxMSP patch, where the data was processed to control soundfiles. The real-time mapping of motion data to virtual objects and music is not a new concept (Bevilacqua, Naugle, and Valverde 2001, Dobrian and Bevilacqua 2003). However, use of the MIDI protocol to multicast real-time motion data to multiple sites over the network appears to be a new technique. 4.2 AGMidi The Access Grid~ is an ensemble of resources linked to enable high bandwidth online multimedia collaboration between any number of remote Access Grid nodes located worldwide. At the heart of this system is the Access Grid Toolkit environment. Written entirely in the scripting language Python, the Access Grid Toolkit (AGTk) provides a "service interface" which allows the programmer/artist to enhance the capabilities of the Access Grid. Included as part of the standard Access Grid install are audio, video, whiteboard, and chat services. At Ryerson University we saw a need to provide artists with collaborative tools which would extend the Access Grid to support work in music, dance, theatre, and new media. To this end we have developed the open source AGMIDI to enable artists to transmit MIDI signals over the Access Grid. AGMIDI consists of a pair of Access Grid services which act much in the same way a hardware MIDI cable would. The sender service takes MIDI data from any MIDI device connected to the computer and uses UDP Multicast to send this data out into the Internet. It is then possible for anyone in the same collaborative space as the sender, to open up an instance of the receiver service, and output MIDI to any connected hardware. Tools such as MIDIYoke ( can also allow receiver nodes to internally loop back MIDI messages to control software on that computer. The sender service also provides an internal loop-back interface to allow performers to monitor themselves. Using UDP Multicast allows data to be transmitted with minimal delay, and to unlimited numbers of receivers without increasing the bandwidth needs of the sender. Thanks to John Harrison's Python port of the cross platform PortMIDI library (, the AGMIDI services can run on nearly any popular platform. The services have been tested on Windows, Apple OSX, and Linux, and all the platforms work seamlessly with one another. Currently the system supports MIDI Machine Control (MMC), MIDI Show Control (MSC), MIDI Time Code (MTC) and General MIDI (GM) messages. Standard MIDI Files (SMF) and eXtensible Music Files (XMF) are currently not supported. The AGTk Service architecture allows the user to launch multiple instances of both the sender and receiver services. On the sender's side, this allows multiple input devices on different hardware interfaces to transmit MIDI simultaneously. Similarly, on the receiver's end, this allows the received MIDI to control multiple devices simultaneously. AGMIDI also allows multiple sending services originating from different physical locations worldwide. The MIDI messages can come from a range of applications such as motion capture systems, lighting control systems, theatrical control systems and of course musical instruments. There are no restrictions on the role of a certain node within the system. A producer node multicasting MIDI can also be a consumer at the same time. This allows for more complex scenarios where one node could produce MIDI signals from a motion capture system, a second node could receive the signal from the network, add their own MIDI data to the stream via MIDI sensors, and in turn retransmit the new multiplexed data-stream to multiple consumer nodes. 4.3 Soundfile Manipulation At the University of Montana, MIDI data translated from the position data of the dancer in the motion-capture system, was received through a multicast network by the AGMIDI service running on a Linux computer, and sent through a MIDI interface to a MaxMSP patch running on a PowerBook. In a later version of the piece, the AGMIDI service ran on the same PowerBook as the MaxMSP patch, simplifying the hardware requirements. In the MaxMSP patch, the y-position of each wrist, mapped to the velocities of two MIDI notes, were used to select between five looping soundfiles. As y-position changed, MIDI velocities moved through five value ranges, representing the five soundfiles. The x-position of each wrist, also mapped to the velocities of two MIDI notes, selected samples within the range of the entire soundfile. Each soundfile was a recording of an InterPlay participant reading a poem written by Nadja Masura, the same poem used for the animation controlled by the motion-capture dancer. The result was that as y-position changed for the right wrist, the x-position of the right wrist determined the starting sample for the corresponding looping soundfile, and as the y-position changed for the left wrist, the x-position determined the ending sample. The effect was real-time granularization of the five looping soundfiles, with the number of samples in the contiguous grains determined by the distance between wrists of the dancer. This chorus of spoken text was mixed in Montana with the sliders of a Peavey PC1600x MIDI interface, controlling the signal level fader objects in the MaxMSP patch, which attenuated the output of each of the soundfiles looping in the 440

Page  00000441 groove- objects. The sliders were also used to control comb-filtering feedback and delay values, varying the level and pitch of the resonant effect. The buttons were used to turn on the comb-filtering, as well as trigger a rhythmic random transposition of the looping soundfiles and the synthesis of FM bell tones, which complimented the resonant comb-filtering. The level of the FM bell tones was controlled with another slider of the MIDI interface. 5 The Void in the Corner and How Many Are We? For scenes one, The Void, and four, How Many Are We?, Scott Deal performed two percussion solos, one composed by himself in 1999 for percussion and electronics, titled Solar Wind, and the other composed by the Australian composer Martin Wesley-Smith in 1983, titled For Marimba and Tape. These pieces were selected as accompaniment for their respective scene, because of how they contributed to the atmosphere comprised by the other dramatic elements. Solar Wind was commissioned for the opening ceremonies of the International Arctic Research Center, a research group funded jointly by the United States and Japan. IARC researches phenomena particular to arctic and sub-arctic regions in both hemispheres, including the study of animal life, ocean physics, and global change. In the area of global change, much of the data collected by IARC scientists is processed in the Earth Simulator, currently the world's largest and fastest computer located in Yokohama, Japan. The Earth Simulator creates models of the oceans and atmosphere which are used to identify trends and to make forecasts of future conditions. Solar Wind was inspired by the phenomena of the same name that creates the Aurora Borealis, or Northern Lights. Actual solar wind consists of ionized particles emanating from the sun that gather at the magnetic north and south poles. When viewed from space, the Aurora is always present as two halos adorning the earth. Sun spots on the surface of the sun can provoke intense solar wind that crashes into the earth creating weather fluctuations and disrupting electronic communications on a global scale. The composition is a study in action-reaction between the tape and live player. igure 5. Charles Nichols performing The Bi as seen at the University of Montana The resultant music for this scene was a structured improvisation, in which the dancer started by exploring the boundaries of the system, as represented by the entire soundfile playing, and gradually moved the starting and ending points of the looping soundfile closer together. When the contiguous grains of samples were small enough to produce a percussive effect, the rhythmic random transposition was triggered with the buttons on the MIDI controller, to produce a rapid variation in pitch. As the piece progressed, more voices were added, and the sounds of different sections of the text were explored by the dancer, as was the effect of crossing wrists, which resulted in the soundfile looping backwards. As the piece grew more cacophonous, the comb-filtering was added, the level of resonance increased, and the pitch of the filtering changed to produce chords, built upon a fundamental drone. Rapidly moving the slider that controlled the delay time of the filters, produced a widely varying vibrato, which added to the building intensity of the piece, as did the addition of synthesized FM bell tones. The piece subsided with the slowing of the dancer's movements, the gradual attenuation of the filtering, and the eventual fade out of each individual voice. Figure 6. Scott Deal performing For Marimba and Tape as seen at the University of Alaska Fairbanks 441

Page  00000442 For Marimba and Tape has become one of the landmark pieces for percussion and tape since its premiere over twenty years ago. It was composed in the Electronic Music Studio of the New South Wales State Conservatorium of Music in Sydney Australia, using a Fairlight CMI (Computer Musical Instrument) which was designed and built in Sydney. The tape part was realized using Fairlight's Music Composition Language. Some of the sounds used were electronically generated, while others came from digital samples of a real marimba. The piece also exists in versions for clarinet and tape and bass-clarinet and tape. 6 The Air Inside Our Head In the third scene, The Air Inside Our Head, Charles Nichols composed and performed a piece for electric violin and interactive computer processing, of the same title. The piece explores the idea of resonant and reverberant space, processing the electric violin with octave doubling, multiple delays, pitch shifting, and flanging, while moving through quadraphonic space, in interactive programming realized with MaxMSP. Because of bandwidth limitations, the piece was broadcast in monophonic sound, for the Interplay. 7 Performance Summary The Interplay performances are opportunities to exercise the Access Grid's strengths and wrestle with its weaknesses, and in the process learn much about performing in cyberspace. While there are the familiar technical challenges of audio-visual quality and bandwidth, as well as issues of signal coordination and latency, perhaps it's greatest weakness currently is in the quality of sound over the Access Grid. In spite of these challenges, the artistic experience of Interplay was compelling because of the conceptual enormity of the medium. Telematic performances are seemingly ubiquitous. Instead of a physical location that is "somewhere," there is a web address that is "everywhere." Anyone can venture there through their computer, and when they do, the minds and senses of real performers, audience and community, are there also. In this virtual nexus, ensemble members hail from a spectrum of art mediums and viewpoints. In the past, cyberspace has been most commonly used by gamers with avatars (virtual characters manipulated via controller), as a training environment for flight, or other kinds of simulation. Now, for artists armed with software, mixers, web cams, 3D environments and other objects in the telematic realm, it is an open territory for collaboration. Because of this sense of the telematic realm being a wide open frontier, it is interesting to wonder how it will develop in coming years. For instance, how will telematic art shape musicians who use it as a performance tool? Perhaps it will further erase the lines between composer, performer, technician and producer. Or perhaps it will bring a broader definition to the term "instrumentalist," so that the essence of certain instrumental families could be distilled into virtual counterparts. In this scenario a cyberpercussionist would be a craftsman of noise, who has mastered a repertory of body gestures that triggers sounds in a 3D environment sophisticated enough that each performer's motions create their own unique sound, and a virtual violinist could control a remote instrument, hearing and feeling its response over the network. Many questions remain, not the least being how important will geographic (therefore, cultural) location remain as the web grows in its sophistication and universality? How egalitarian or exclusive will tomorrow's Internet be for emerging musicians and artists? In a medium where the tools evolve on the average of every 18 months, what aesthetic standards will define great works that stand the test of time? 8 Future Work Rehearsals are underway for a new Interplay developed by the Another Language Performing Arts Company, entitled Dancing on the Banks of Packet Creek, which again includes live musical performance as accompaniment to the other dramatic elements. Other plans include using AGMidi to control animation as well as music, with MIDI data sent between multiple sites, and experimenting with broadcasting multiple channels of audio. 9 Acknowledgments Thanks to all of the artists and technicians at the Universities of Alaska, Illinois, Maryland, Montana, Purdue, Ryerson, and Utah, who participated in this project. References Bevilacqua, F., Naugle, L., and Valverde, I. 2001. Virtual dance and music environment using motion capture. In Proceedings of the IEEE Multimedia Technology And Applications Conference, Irvine, CA. Dobrian, C. and Bevilacqua, F. 2003. Gestural control of music using the vicon 8 motion capture system. In Proceedings of the New Interfaces for Musical Expression Conference, Montreal, Quebec, Canada. Meador, W. S., Rogers, T. J., O'Neal, K., Kurt, E., and Cunningham, C. 2004. Mixing dance realities: Collaborative development of live-motion capture in a performing arts environment. ACM Computers in Entertainment (CIE), 2(2), 1 -15. 442