Page  1 ï~~COMPOSING FOR RELATIVE TIME IN DISTRIBUTED PERFORMANCE Robert Rowe New York University Steinhardt School Department of Music & Performing Arts Professions ABSTRACT Music is a temporal art. Recent experience with distributed performance has introduced new challenges to traditional ways of thinking about time with respect to ensemble composition and performance. Performing across two sites connected by the Internet entails compensating for delays, though those delays have become shorter and musicians more experienced in dealing with them in recent years. When the number of sites involved grows to three or more, however, another layer of complexity is added as performers at the various sites are confronted with different configurations of latency, and thereby significantly different performances, at each location. 1. INTRODUCTION Distributed performance has been with us for many years now [3][4]. In this paper I will discuss in particular those performances that are distributed via the Internet, in which one or more venues broadcast to and receive audio and video signals from the other sites and render these onstage as part of the performance itself. At New York University we recently participated in a demonstration for the Audio Engineering Society conference of a communications protocol and compression scheme that was able to connect musicians in New York and McGill University in Montreal, via audio and video signals, with under 20 milliseconds of latency [1]. A similar demonstration between NYU and McGill in 1999 had 1000-3000 milliseconds of latency [5]. The difference between even 1000 and 20 ms of latency is so profound as to constitute a different technology altogether. Twenty milliseconds of delay is the same as that experienced by two musicians seated twenty feet apart. In the AES conference, a drummer in Montreal and a bass player in New York were able to play rhythmic material in perfect synchrony. With 1000 milliseconds of latency, such an outcome cannot even be attempted. 2. LATENCY TOLERANCE Transmission rates over the Internet range from a 28.8 thousand bits-per-second (bps, or baud) telephone dialup line to the 100 million bps or more available on high-speed broadband connections. Clearly, a 28.8k baud telephone connection is too slow to keep up with even the 31.25k baud bandwidth requirement of MIDI, not to mention the 1.4 megabaud needed for stereo, CDquality digital audio. Even when a transmission channel with sufficient theoretical bandwidth is used, signals going into and coming out of the link are often buffered to compensate for network congestion between the two machines. Depending on the nature of the signals being sent and the quality of the transmission channel, these buffers may typically range anywhere from 5 to 1000 milliseconds or more. Given that background, what degree of latency is tolerable in a distributed performance situation? In a recent paper titled "Effects of time delay on ensemble accuracy", Chris Chafe and colleagues at Stanford report on tests of rhythmic performance under varying latency conditions [2]. They asked pairs of musicians located in separate rooms but with audio shared through headphones to clap a simple rhythm together. Increasing delays beyond a duration that might normally be encountered in live performance led to an eventual breakdown in the shared rhythm. The observed behavior was a gradual tendency toward slowing down the tempo as delays increased, but eventually that strategy could no longer successfully maintain the rhythm and the performance would break down altogether. The ideal delay for performing the rhythm in a steady tempo was 11.5 milliseconds - with shorter latencies performers actually tended to speed up slightly. "The observed optimal one-way delay dbest = 11.5 ms equates with a physical radius of 2,400 km (assuming signals traveling at approximately 70% the speed of light and no routing delays)" [2]. Though the ideal is 11.5 ms, the performance falls off slowly enough to indicate the possibility of reasonable performance up to 20 ms or so. 3. THE TECHNOPHOBE & THE MADMAN In 2000-2001 the New York State Council on the Arts (NYSCA) commissioned a work from New York University, Rensselaer Polytechnic Institute, and Harvestworks (a New York-based arts foundation) that would be presented at NYU and RPI simultaneously, using an Internet connection between the two sites. The purpose of this commission (which became The Technophobe and the Madman) was to explore the artistic consequences of composing for the medium of Internet2. One expression of this exploration was an extensive rehearsal period that spanned several months

Page  2 ï~~of trials across a connection between the two sites. Repeated encounters with the realities of distributed performance had a profound effect on the piece as it was being developed. The Technophobe project required guaranteed, uninterrupted connectivity of approximately 20 Mbps (megabits per second) between Troy, New York and New York City for several rehearsals and the concert. In keeping with the general manner of operation of Internet2, the delivery of that connectivity was arranged by direct contacts with systems administrators along the entire data path. The data sent between the two sites consisted of three channels of video information in each direction, and six channels of audio each way, for a total of 6 video and 12 audio channels transmitted within the 20 Mbps connection. The realization of these connections was handled by a commercially available codec, the Vbrick 3000. 4. DISTRIBUTED PERFORMANCE Moma, Figure 2. Proposed full routing configuration for the World Opera Project. 5. CONCLUSION Distributed performance has emerged viable art form in the past twenty years. as a vital and Figure 1. New York stage configuration for The Technophobe and the Madman Primarily due to the video compression handled by the Vbricks on either end of a connection, latency between the two sites averaged 250 ms (one-way). As that delay is well above the threshold for ensemble synchrony, the composers (Nick Didkovsky, Neil Rolnick, and Robert Rowe) had to devise and score several strategies for performance of the distributed musical parts. One of these was the concept of "floating progressions:" chord sequences that indicated which site should initiate a change of harmony. Normally the two sites would alternate in making a harmonic change, and when the new chord was heard by the other side, they would follow. Alternation avoided a karaoke-like effect in which one venue was always reacting to the other. Another approach was used for those sections in which a steady beat was needed: one site would lead, and the other follow, but the first site did not hear the contribution of the second (as it would confuse the rhythm). Note that today, with much faster connection speeds, much of this would become superfluous: NYU to RPI is well within the geographic limits required for perfect synchrony as demonstrated by the Stanford experiments [3]. 6. REFERENCES [1] Cardt, A., Jens, H., Kraemer, U., Schuller, G., Wabnik, S., & Werner, C. "Network music performance with ultra-low-delay audio coding under unreliable network conditions", Proceedings of the 123rd AES Convention, New York, 2007. [2] Chafe, C., Gurevich, M., Leslie, G., & Tyan, S. "Effect of time delay on ensemble accuracy," Proceedings of the International Symposium on Musical Acoustics, Nara, Japan, 2004. [3] Renaud, A., Cardt, A., & Rebelo, P. "Networked music performance: State of the art", Proceedings of the 30th AES International Conference, Saariselkii, Finland, 2007. [4] Rowe, R. "Real and unreal time: Expression in distributed performance", Journal of New Music Research 34:1, 87-95, 2005. [5] Xu, A., Woszczyk, W., Settel, Z., Pennycook, B., Rowe, R., Galanter, P., & Bary, J. "Realtime streaming of multichannel audio over Internet," Journal of the Audio Engineering Society 48 (7/8): 627-641, 2000.