Page  202 ï~~Design and Implementation of a Multimedia Opera James Oliverio Georgia Institute of Technology Department of Music Director, Audio Research Team j ames.oliverio @ music. Jarrell Pair Georgia Institute of Technology Department of Music Assistant Director, Audio Research Team jarrell.pair @ Abstract While the traditional notion of "opera" describes a production that incorporates musical elements with other art forms and media, the conceptual and technical considerations required to mount an electronic multimedia opera add significant layers of complexity. Design issues including projecting visual media, the implementing of an immersive audio environment and maintaining synchronization (between live singers and digital music, 3D computer animation, stage lighting and projected imagery) add to the standard list of compositional and production challenges. An overview of the design and implementation of such a production, StarChild: The Opera, offers a primer in the musical and extra-musical considerations that are required to realize a large-scale composition in an electronic mixed media environment. 1 A New Approach to Opera The production of traditional opera is often based upon fairly standardized musical and dramatic staging techniques. Until recently, the vast expressive and creative potential of electroacoustic music technology remained largely unexplored in opera composition and performance. However, precedents exist for new forms of opera. One of these new forms, multimedia opera, draws upon music technology to achieve new combinations of artistic resources. The term "multimedia" in this context does not refer exclusively to interactive media. It is also inclusive of productions which use a mixture of audio and visual formats in their structure and content. As Miroslaw Rogala and Darrell Moore explain, multimedia is nothing more than "... plural art forms operating in conjunction with one another" [Rogala and Moore, 1993]. Steve Reich's The Cave and Tod Machover's Brain Opera are two recent examples of this new form of opera. In order for multimedia opera to become more prevalent, creative and technical personnel need guidelines to follow for production and composition. The production of StarChild, a multimedia opera by James Oliverio, provides a valuable guide for those planning to create this type of musical production. 2 Use of Electroacoustic Music Technology Multimedia opera utilizes a broad pallet of sound and light to enhance its presentation. A careful balance must be achieved to serve the music and plot without overcomplicating the technical demands of the production. StarChild utilized current music technology in a variety of ways. All of the preproduced audio components were digitally recorded and edited. Additionally, electroacoustic music technology was used to provide solutions for immersive audio, data sonification, and device synchronization. Each of these items was implemented based on its suitability to the opera's content. 2.1 Immersive Audio In StarChild, we wanted to give the audience the sensation that it was surrounded by characters conversing telepathically across the vast distances of interstellar space. Standard two channel stereo audio was deemed ineffective for this production. We did not have access to industry prevalent surround audio encoding and decoding schemes. As an alternative, we chose to create a custom GUI based system to steer multiple audio sources among eight speakers. The system was first prototyped using a modified joystick which could steer one source among four speakers. Next, a PC based system using external DSP hardware was assembled to steer multiple sources. The system operates by first loading directional information files onto the external DSP hardware. The DSP hardware uses these files to steer digital audio across multiple output channels as it is played back. The steered audio is processed using a Digidesign Protools ImI system and recorded on a DA88 master tape. We plan to use this system as a generic method for efficiently creating multichannel surround audio. During late production, we Oliverio & Pair 202 ICMC Proceedings 1996

Page  203 ï~~considered creating true 3-D audio effects by using ambisonic techniques as described by Malham and Myatt[ 1995]. However, time constraints prevented us from exploring this option. 2.2 Data Sonification In recent years, data sonification has been seen as a tool with great potential for studying complex sets of scientific data. Basically, this analysis method maps data to sound parameters. Sonification can provide the researcher with the ability to perceive variations and trends that are invisible to more established data analysis techniques. Madhyastha and Reed[1995] demonstrated that a sonification system can be realized using a computer capable of playing digital audio samples or triggering external sound modules. However, few have explored the potential artistic applications of sonification. A notable exception is seen in Music From the Galaxies, a compact disc of compositions by Dr. Fiorella Terenzi based on astronomical data. We saw StarChild as an excellent opportunity to explore sonification's potential artistic applications. Using institute and public resources, data was collected from the comet Shoemaker-Levy 9's collisions with Jupiter. This data coincided with the astrophysical aspects of the opera's story. We used Matlab software and Csound to take floating point data from the collisions as input for creating music and audio effects. Additionally, images of the Shoemaker-Levy 9 comet collision with Jupiter were transduced into audio effects using Hyperupic, a NeXT computer application by Christopher Penrose. Experimentation with different data sets and parameter settings was necessary to produce aesthetically pleasing audio content. 2.3 Music Synchronization Methods Various tape and computer systems had to be coordinated to achieve timing effects with split-second accuracy. MIDI can be effectively used to control visual effects[McLean, 1992]. Yet, it was decided that SMPTE time code would be the most efficient way of coordinating these various systems, which included 2 Beta video decks, 15 slide projectors, computer-controlled lighting and a DA-88 digital multitrack. In complex synchronization efforts, a multitude of difficulties arise. McLarnon and Muxie[1994] provide a description of how to prevent the more commons problems. While the singers performed live, all of the music and a great deal of the dramatic dialogue and monologue was delivered from the digital multitrack through the eight channel sound system. Whereas over eighty percent of the opera had visuals that were projected from either video or slide sources, it was determined that the master time code source would be the Beta video deck 1. In a very real sense, the animation "directed" the flow of the opera, so that visual and musical synchronization would always remain intact. 3 Conclusions Opera singers are traditionally at the mercy of the conductor's tempi and the orchestra's intonation, not to mention the composer's plight. This dilemma is nonexistent in this form of opera. The composer was able to work directly with the vocal artists to determine not only the best key and tempo for each aria, but also to sculpt musical phasing via MIDI files before making the master recording. Hence the artists always knew exactly what musical parameters they would be dealing with. They could prepare for the world premiere performances with portable cassette recordings of the actual instrumentation, phrasing and tempi that they would rely upon during the live performances. Consequently, the composer was assured that his intention would be realized in the world premiere performance. The aesthetic depth allowed by pre-producing dialogue and sound design with the computer animation has truly opened up a new direction for musical theater, one that was previously the sole province of film and television producers. Given the influence of cinema on twentieth and twenty-first century viewers, these influences can only help to draw and maintain audiences for new offerings growing out of the musical form traditionally known as "opera". References [Madhyastha and Reed, 1995] Tara M. Madhyastha and Daniel A. Reed. Data Sonification: Do You See What I Hear? IEEE Software, 12: pp.45-56, 1995. [Malham and Myatt, 1995] David G. Malham and Anthony Myatt. 3-D Sound Spatialization using Ambisonic Techniques. Computer Music Journal, 19(4): pp.58-70, 1995. [McLarnon and Muxie, 1994] Zed McLarnon and J.P. Muxie. Sync Problems Faced by the Multimedia Producer Now. Advanced Imaging, 9(1): pp.62-64, 1994. [McLean, 1992] Barton McLean. Composition with Sound and Light. Leonardo Music Journal, 2(1): pp. 19-22, 1992. [Rogala and Moore, 1993] Miroslaw Rogala and Darrell Moore. Nature is Leaving Us: A Video Theatre Work. Leonardo, 26(1 ): pp. 11- 18, 1993. 1CMC Proceedings 1996 203 Oliverio & Pair