Page  00000001 LARGE AUDIENCE PARTICIPATION, TECHNOLOGY, AND ORCHESTRAL PERFORMANCE Jason Freeman Georgia Institute of Technology Music Department ABSTRACT Glimmer, a composition for chamber orchestra and audience, uses novelty light sticks, video cameras, computer software, multi-colored stand lights, and projected video animation to create a continuous feedback loop in which audience activities, software algorithms, and orchestral performance together create the music. This paper establishes the aesthetic background and motivations behind Glimmer, describes the conceptual and technical design of the piece, and evaluates Glimmer's successes and failures in meeting its design goals. 1. INTRODUCTION Glimmer, a composition for chamber orchestra, engages the concert audience as musical collaborators who do not just listen to the performance but actively shape it. Each audience member is given a battery-operated light stick which he or she turns on and off over the course of the piece. Computer software analyzes live video of the audience and sends instructions to the orchestra via multi-colored lights mounted on each player's stand. Typically, audiences at orchestral concerts have little interaction with the performers and composers of the music they hear. They sit quietly in their seats, watching a conductor whose back is turned towards them, and they must wait until the music is over to respond with their applause. By that time, of course, it is too late for their response to affect the performance they just heard. They are passive spectators of an event, not participants in it. With Glimmer, I was interested in reimagining these boundaries in order to create a more collaborative musical experience and to encourage audiences to discover their own creativity as they listen in new ways. 1.1. Musical Precedents Glimmer follows in the tradition of musical works which facilitate real-time participation by a large audience during their performance. In many such works, audience members become performers, creating some or even all of the music. For example, in Jean Hasse's Moths (1986), the audience whistles as directed by a conductor and a graphical score to perform the piece [7]. During La symphonie du millinaire (2000), an outdoor performance event in Montreal, 2000 audience members rang handheld bells at designated times [3]. And many Fluxus scores specify or imply more open-ended audience participation, as with Tomas Schmit's Sanitas No. 35 (1962): "Blank sheets are handed to the audience without any explanations. 5 minutes waiting" [11]. In other works, the audience contributes input which affects the musical performance, rather than creating sounds which are part of the performance. For example, in Thomas C. Duffy's The Critic's Choice (1995), a film-music takeoff for concert band, the audience votes for one of three alternate endings [4]. And in a performance of Terry Riley's In C (1964) staged by the Eos Orchestra, audience groups seated at tables tapped electronic dome-shaped centerpieces to advance a MIDI instrument to the next musical motive [1]. A final category of projects use technology to engage a live concert audience in new ways while retaining the audience's fundamentally passive role. Golan Levin's DialTones: A Telesymphony (2001) creates music by triggering audience mobile phones to play pre-composed ringtones [12]. And the Concert Companion provides real-time program notes about orchestral repertory via wireless PDAs [9]. 1.2. Gaming Precedents Glimmer is also inspired by mass-audience games which use technology to enable large audiences to participate without leaving their seats. In projects developed by Cinematrix, audience members hold up the red or green side of a paddle to collectively navigate objects on a video screen [2]. More recent systems have used video tracking of audience members as they shift left and right in their seats [8] and motion tracking of giant weather balloons which circulate through the seating area [10] to facilitate similar types of interaction. 1.3. Goals Glimmer follows in the spirit of the Duffy and Eos works and the gaming examples; I wanted the audience to play the role of a Greek chorus rather than of the chorus at a Messiah "Sing-In." Audience activities influence the actions of the orchestral musicians on stage rather than directly creating the sounds of the piece. I did so in order to maintain some separation between audience and orchestra and to make nonmusicians and musicians alike comfortable in participating. I also wanted to give the audience an opportunity to contribute not merely surface content to the work, nor to simply choose from a limited menu of pre-conceived paths, but to influence the work at a lower level, affecting even which notes were played by which players at which times.

Page  00000002 It was also important that the system be conceptually simple. The realities of the concert environment - limited rehearsal time, limited time to train the audience, and a contractual limit for the piece to be only ten minutes - made this a necessity. Finally, as with Tomas Schmit's piece, I did not want to direct audience actions via any kind of predetermined score. Rather, I wanted to create an environment for them to explore and an opportunity for group behaviors to emerge. 1.4. The Role of Technology Glimmer uses technology as a means to facilitate collaboration. Hardware and software are used to translate audience input into musician instructions, quickly performing analysis, decision-making, and communications tasks which would be impossible for a human or group of humans to do in such a short amount of time. But all the music is generated acoustically by the musicians onstage; there is no computer-generated sound. 2. SYSTEM DESIGN Glimmer is designed as a continuous interactive feedback loop. Video images of audience activity are analyzed by computer software. The analysis data is transformed by control software into performance instructions for the musicians and video animation for the audience. The musicians play music based on their instructions. The audience reacts to the music they hear and the video they see, changing their activities and initiating another iteration through the loop. I ars ' o^ m o r m u r i --- --- sys a---- J-..............,.'. i.,1i. (,7.. I.. Four consumer-grade video cameras capture images of the entire audience and forward them to a video computer for analysis. Computer software, written with Cycling '74's Max and Jitter, pre-processes each frame, performing color plane extraction, image masking, threshold noise reduction, and image dilation and erosion. It then determines the percentage of audience members in each group whose light sticks are activated, counting blobs of adjacent non-black pixels and dividing the number of blobs by the total number of people in the group. The resulting percentages are forwarded to a control computer. 2.2. Control Computer Computer software, written with Cycling '74's Max, translates incoming audience data into outgoing musician instructions and video animation. 2.2.1. Direct Mapping On a basic level, the light-stick activation percentage for a group controls the dynamic at which that group's musicians play. If everyone in a group has their light sticks turned on, their group plays as loud as possible. If everyone has them turned off, the group is silent. 2.2.2. Competitive Mapping On a higher level, there is a comparative analysis of the activities of all seven audience groups. Groups which generally have faster rates of change in their light-stick activation are rewarded: their musicians are more likely to play, they play at a higher dynamic, and they change pitches more often. The software continuously updates group rankings (from first through seventh place) and uses these rankings to determine which texture is assigned to each group (see below). Groups which are ranked higher are also mapped onto a wider dynamic range in the direct mapping. When a group jumps into first-place position, it cues a change in its pitch or pitches accompanied by an accented attack. 2.2.3. Textures Throughout the piece, each group sustains single notes or clusters of notes which gradually cross fade from one musician to the next: one player will decrescendo to niente as another player crescendos from niente. Figure 2. Visual representations of two different textures used in the piece. The software defines several different variations on these textures in which the number of simultaneous sustained notes, the total set of available pitches, and the Figure 1. Glimmer's interactive feedback loop. A brief overview of the hardware system and software algorithms follows; interested readers should refer to the score for additional details [6]. 2.1. Audience Input and Video Analysis Each audience member uses a four-inch long batteryoperated LED light stick during the performance, switching it on or off by twisting its cap. The audience is divided into seven groups; each group contains approximately 75 people and controls a corresponding group of three or four musicians in the orchestra.

Page  00000003 speed of cross fading all vary. A lookup table maps group ranking to assigned texture. 2.2.4. Musical Structure The ranking-to-texture lookup table changes over the course of the piece, giving the music a large-scale structural shape. Each individual change to the table is barely perceptible, so that on a local level, audiencedriven events take perceptual precedence over precomposed cues. The large-scale arch form begins with just a single group playing a single note at a time, gradually becoming denser until all seven groups are playing clusters of notes chosen from a 28-note diatonic set. The closing minutes of the piece are a cross between Haydn's "Farewell" symphony and a reality television show, as groups are removed from the piece one by one based on their cumulative competitive rank. The music itself is extremely simple, as sets of pitches and timbral combinations are constantly but gradually transformed; works such as John Cage's Four2 for chorus (1990) were influential. This helps audience members easily identify their own group within the orchestra and also counterbalances the novelty of the interaction. 2.3. Multi-Colored Lights and Video Projection The orchestral musicians do not read from conventional musical notation nor do they follow cues from a conductor. Instead, each player receives real-time instructions from the computer via a Color Kinetics iAccent multi-colored light, which sits on his or her music stand. Each light is controlled independently and changes color continuously. The color family of a musician's light - brown, green, blue, or pink - indicates which of four notated pitches to play. The brightness of the light indicates the dynamic at which to play. Short flashes of light prepare musicians for note changes and accents. A simple video animation, projected onto a screen behind the orchestra, helps the audience more easily follow the relationship between their activities and the music they hear. Each audience group, represented by a rectangle, changes color based on the group's activation percentage and competitive rank, and the first-place group receives additional visual emphasis. 2.4. Reliability Within the Orchestral Environment Decisions about the system design and its technical implementation were heavily influenced by the limitations of contemporary orchestral performance environments. Limited rehearsal time and access to the hall, combined with a shoestring budget and the impossibility of rehearsing the piece with a full audience, all made flexibility and reliability top priorities. Even more than the redundant backup computers, the runtime control and monitoring tools, and the testing and simulation components, the Color Kinetics iAccent lights were the most dramatic example of the reliability measures taken. These water-resistant lighting units, certified for 100,000 hours of outdoor operation, communicated with computer software via industrystandard Ethernet protocols. They also facilitated quick stage setup, as they could be daisy-chained together with combined power-data cables in any order. When given the choice between hacking together a proprietary solution or leveraging an industrial-grade product, the answer was obvious. And in fact, renting these lights was no more expensive than building a custom solution. 3. DISCUSSION The American Composers Orchestra asked me to write a piece which used technology and was fun: in these respects, the premiere of Glimmer was a tremendous success. The audience enjoyed their role, gasping and laughing at moments of surprise and drama during the performance. They also spontaneously developed creative ways to participate, including a version of the ballpark wave in which light sticks were dramatically raised and lowered to show and hide them from the camera's view. And the hardware and software performed nearly flawlessly; the largest problems were human rather than mechanical. For example, the video projector was inadvertently left off during the dress rehearsal. And one of the violinists was color blind. 3.1. Audience Participation Early in the development of Glimmer, a colleague asked me how I would evaluate the piece's success. I responded that if every audience member believed that the performance would have been different without him or her, then I would be satisfied. While some audience members did feel that way, recalling specific moments where they made a noticeable difference, others were frustrated that none of their actions seemed to matter. Since Glimmer's algorithms respond to the activity of entire audience groups rather than of individual members, a large part of the problem lay in groups' inability to work together to influence the performance. When many group members switched their lights on and off quickly - but out of sync with their neighbors - their activities simply cancelled each other out. As a result, the on-off percentages of groups varied by a disappointingly small amount over the course of the performance. My hope had been that even in the absence of group leaders, interesting group behavior would emerge over time in a manner similar to cellular automata. The simple rules which governed the competitive aspect of the piece were designed to encourage such behavior, but while the competition added an exciting dimension to the experience, it failed to accomplish its original goal. In informal discussions with audience members, I learned of several reasons why groups had failed to cohere. Some people complained that the piece was too short for them to develop a group sensibility; they felt

Page  00000004 they would have done better had the piece been longer, or had it been performed a second time. Others had trouble seeing all the people in their group, so it was difficult to respond to what peers were doing. But most importantly, audience members enjoyed waving their light sticks around much more than switching them on and off, even though they knew that such activity had little effect on the music. Not only was it more fun to do, and not only was it more pleasing to watch, but it also gave them the feeling of more communication with and control over their peers. They were able to communicate a range of information to each other - if not to the computer software - through their stick's position and speed, going beyond mere onoff signals. 3.2. The Role of the Orchestra In Glimmer, there is a fundamental inequality between the audience and the orchestra. The audience works within the framework defined by the piece but follows no score, while the orchestra closely follows instructions and has only limited interpretive freedom. While I was enticed by the idea of giving the orchestral musicians a greater interpretive role, it did not make practical sense in Glimmer. The orchestral musicians struggled to familiarize themselves with their lighting cues during the single hour of rehearsal. It would have been difficult to ask classically-trained musicians to also learn to make unfamiliar interpretive decisions within that time frame. Furthermore, the music is constructed so that perceptually salient local events always originate from audience activity. Were musicians to alter these events or add their own, it would be much more difficult for audience members to establish the relationship between the things they did and the music they heard. 3.3. Final Thoughts There are some simple changes which could improve Glimmer's interactive experience, such as analyzing different audience activities, providing clearer visual feedback through the video projection, and restructuring the verbal introduction to the piece to include interactive practice opportunities. As other creators of large-scale interactive works have also found, convincing the audience that they have control and teaching them how to exercise it is a large part of the challenge [5]. But could the work ever make all 600 audience members feel truly indispensable to its performance? Large-audience participatory works cannot promise instant gratification: giving each person a critical role; requiring no degree of experience, skill, or talent; and creating a unified result which satisfies everyone. Works such as Glimmer reveal the impossibility of this goal even as they strive towards it. They invite participants to explore an environment, discover its limits, and find imaginative ways to express their creativity by pushing against those limits. 4. ACKNOWLEDGEMENTS Glimmer was commissioned by the American Composers Orchestra, Steven Sloane, music director, Robert Beaser, artistic director, Dennis Russell Davies, conductor laureate. It was premiered on January 21, 2005 at Zankel Hall at Carnegie Hall in New York. This work was funded in part by the Composer Assistance Program of the American Music Center. Additional thanks to Akademie Schloss Solitude, the Columbia University Computer Music Center, and Meet the Composer's Creative Connections program. 5. REFERENCES [1] Bianciardi, D., Igoe, T., and Singer, S. "Eos Pods: Wireless Devices for Interactive Musical Performance." Proceedings of the Fifth Annual Conference on Ubiquitous Computing, Seattle, 2003. [2] Carpenter, L., and Carpenter, R. "Audience Participation." Ars Electronica: Facing the Future, ed. T. Druckrey. Cambridge: MIT Press, 1999. [3] Chenard, M. "The Millenium Symphony: A Work for the Beginning of Time; Part I: The Musical Challenge." La Scena Musical, Vol. 5, No. 8, May 2000. [4] Duffy, T. The Critic's Choice (score). Selfpublished, New Haven, 1995. [5] Fisher, R., Vanouse, P., Dannenberg, R., and Christensen, J. "Audience Interactivity: A Case Study in Three Perspectives." Proceedings of the Sixth Biennial Symposium for Arts and Technology. Connecticut College, 1997. [6] Freeman, J. Glimmer (score). Available at http://www.jasonfreeman.net, 2005. [7] Hasse, J. Moths. Visible Music, Euclid, OH, 1986. [8] Maynes-Aminzade, D., Pausch, R., and Seitz, S. "Techniques for Interactive Audience Participation." Proceedings of the IEEE International Conference on Multimodal Interfaces, Pittsburgh, 2002. [9] Mirapaul, M. "A Hand-Held Portal to Musical Delights." The New York Times, July 17, 2003. [10]NYU Movement Group, SquidBall, http://movement.nyu.edu/commotion/squidball, 2004. [11]Schmit, T. Sanitas No. 35. Available at http://www.artnotart.com/fluxus/tschmitsanitasno.35.html, 1962. [12] Sheridan, M. "Composer Calling: Cell Phone Symphony Premieres." NewMusicBox, American Music Center, New York, Oct. 2001.