Page  00000338 O PE N DRA~MA Networking Opera: a report of a project in progress Flavio Tariffi, Stefano Cuomo Space S.p.A., Italy Carola Boehm, Nick Bailey, Phil Kerr, Barry Short Centre for Music Technology, Univ. Glasgow, UK Abstract Opera is one of the most universally known and appreciated forms of art. Its heritage encompasses four centuries of creation, performance, criticism, emulation and - in modem time - analogue and digital recording on vinyl records, CDs, DVDs and video tapes/video discs. Despite the wide availability of audiovisual reproductions, though, no significant result has yet been attained in delivering the magic of opera to users in deeply interactive, creative and entertaining ways, by making use of multimedia technology and extending beyond the constraints of off-line media. The failure of dedicated hybrid media such as the ill-fated Enhanced CD or CD Extra has further constrained a potentially huge market, that is now ready to endorse a new generation of high-quality network services encompassing both rich-interactivity broadcasts and sophisticated "opera -on-demand" personal and group experiences. The aim of the project is the development and integration of advanced multimedia solutions in order to provide new enhanced user experience in the dynamic interaction with the heritage of lyric opera and other vocal dramatic music (cantatas, oratorios, masses, Lieder, society and entertainment music, etc.). The main objective is to open this unique heritage to a novel dimension of learning, exploring and entertainment, with a special emphasis on real-time and multi-user interaction and on the delivery of rich and adaptive cultural objects on telecommunications networks based on the Internet and its integration with other media. 1. Project Objectives, Consortium and Aims The project OpenDrama started at the end of 2001 and will run for two years with a partner consortium consisting of * Space S.p.A. (Italy), * System Simulations Ltd (UK) * Centre for Music Technology, University of Glas gow, UK * Politecnico di Milano (Italy) * Universitat Pompe Fabre - Barcelona (Spain) * Europe Online Networks S.A. (Louxembourg) * Dynamic S.r.l. (Italy) * Discoteca di Stato (Italy) * Teatro del Maggio (Italy) * Opera North (UK) In application terms the service envisaged will enable users to interact with historical opera re cordings, mainstream opera on disc and live opera performances in a completely new and very attractive way: * Enabling both passive attendance and random access; * Dynamically explaining the plot and visually showing the relationships between characters, their pre-action stories, etc; * Making the libretto dynamically available in real time; * Showing the music notation in synch with the execution and commenting on notable musical feature s; * Enabling users to "feel the sound" of any part of the partition by simply clicking on the selected measures; * Enabling users to turn parts (voices, instruments) on and off to interactively understand the musical texture; * Enabling users to substitute themselves to any of the singers, singularly and in groups; * Mixing high-quality digital audio with video takes, backstage information, movie materials, biographies of authors and singers, plot accounts, anecdotes and actual/virtual reconstructions of the dramatic scenery; 338

Page  00000339 OPENDRAMA THE MAIN FUNCTIONAL COMPONENTS IOBJECTI CHANNELj USER MacBeth Lady SyVisualisation of < plot and dramatic Banquo MacDuff tructures Interactive karaoke mo Virtual staging with 3D avatars Contextual visualisation of score and libretto The Opera Memory Palace community service * Enabling users to dynamically access the service database in order to make comparisons between different performances of the same scene, to hear famous singers or directors face-to-face or to compare different settings of the same libretto (e.g., Paisiello and Mozart's Barbiere di Siviglia; Schubert's and Wolfs setting of Goethe's Prometheus, and so on); * Providing new versatile tools and contents for didactics in fields such as the history of music, modem literature, modem history, history of cinema, singing, etc.; * Letting users, under the appropriate licensing conditions, make digital recordings of the broadcasts and on-demand transmissions for off-line * Enabling users to interact with a rich community of experts and enthusiasts with the support of a host of online services (chat, records exchange, quiz games, etc.). In technical terms the project will aim at implementing and applying media acquisition, tagging, encoding, storage and delivery solutions enabling the interaction of end users with rich multimedia cultural objects in an easy and satisfactory way. The technological background will be provided by an implementation of MPEG-4 and MPEG-7 technology (subject to licensing considerations), as well providing similar services with open source solutions. 2. First Results As one of the first results of this project, discussions with opera houses, archives and other content owners have and will continue to feed into a report of a detailed socioeconomic requirements study. 2.1. Socio-economic requirements from discussion Initial results for defining the interest from content owners valuing services which OpenDrama could provide are emerging under the following points below: 2.1.1. Ownership of Creative Processes: Opera Houses prefer to have as large as possible creative input and decision making in this as the "creative content creator". The openness to any new ideas for "enriching" this media and coming from the creative producers/directors handling the content is preferred within all phases of production. Producers will/might and are likely b want control over what and when it is being produced, and this will possibly relate to the "traditional performance" in some way, i.e. providing feedback in both directions. This does provide design and logistical production problems, as the usual traditionally web-based projects normally do not have to be closely linked "real-life" production time-lines. It call for a new methodology for production logistics, in which the production preparation logistics of the normal performance are seamlessly 339

Page  00000340 integrated with the production logistics of a "multi-media project. We expect to gain valuable experiences in this project regarding this, which will possibly influence the design of future systems, offering "editing suites" for opera production purposes. 2.1.2. Content Pull, not Technology Push: One major aspect for Opera Houses is that "Content is King". Most issues regarding the handling, the performance, method of capturing and displaying, including distribution, is for Opera Houses inseparably hooked up to the choice of content. The content asks, or demands a certain handling or representation, all other questions need to be considered in the light of a specific content, i.e. a specific opera. Obviously for funding bodies but moreover funding applicants, this can be literal hell, as funding applications tend to demand that the technology is defined in advance and content can be created "seiendipically" while using the technology. The notion of starting from the content, and building the technology around it, and possibly building serendipity into the software development process itself provides financing problems. The few central, and better funded large cultural institutions tend to be used to taking this risk, funding bodies for technological development usually not. This reflects the outcomes of the CIRCUS project, in which there was a high demand for supporting "creative pull" instead of accepting the usual "technology push".1 2.1.3. Efficiency: Integration of the creating of OpenDrama objects within the creation process of the Opera itself: There is a very strong argument for choosing a work for interactive media production which is in the process of making. This maximizes the value and minimizes the costs of processes which tend to be extremely costly. Multimedia productions of this kind might need to learn from vast logistical experience of the Broadcasting community, in which it is usually out of the question to stage an opera solely for broadcast. Hooking up the capturing of Opera related digital media to the whole creative process of the opera production and integrating this into the rehearsal phase, provides additionally opportunities for promotion and dissemination, thus gaining interest even before the first performance. For an emerging production methodology this implies that logistics need to be defined with the aim that the continuous products or services of a "digital opera" and the "traditional opera" reinforce each other throughout the whole rehearsal phase. 2.1.4. Balancing Acts: Quality vs Quantity: Within the logistics of capturing digital multimedia enhanced opera objects, the costing process is a relative complex one which aims to maximize the quality and quantity within limited amounts of budgets. This necessarily means, one has to balance quality material with quantity material, i.e. How many pieces of the opera are being recorded in multitrack I CIRCUS - Content Integrated Research into Creative User Systems, studio sessions with full separation of sound vs how simple will be a recording of the whole opera. The choice of where to reduce the quality and where the quantity must simply be present, as mentioned above, must ultimately derive from the process of considering the content and the creative aim of the production, i.e. it cannot and must be not only a pragmatic financial choice, but a creative one. 2.1.5. Supporting "Friends of the Opera": Opera Houses are very keen to cater for those stakeholders of production, who provide the most stable income for Opera Houses, such as "Friends of Opera", "Patrons", "Opera Circles" or long term "Abonnementiers". Their loyalty is being valued extremely high by Opera Houses. Presently Opera houses provide value-added events for these groups, such as talks and seminars and study days. Within the digital opera, this could be expanded upon with archiving these, as well as adding additional services, which would enable stakeholders to have a look into production process which in turn could provide more insight and information of how their money is being spent. This consecutively could increase the attractiveness of becoming a patron. 2.1.6. Supporting a community: Opera Houses are keen not only to provide new dissemination channels of the medium opera or create new production models to aid new innovative means of creating and distributing the medium opera, but they are also keen to create and widen interested communities for Opera interested individuals, as well as stakeholders of opera productions, by using technology best to support this community. Ideas being toyed with at different Opera Houses include live cameras in rehearsals or other opera places, to which for instance guest directors could be provided access. Apart from internal users, external users could have access to the public face of this. 2.2. The Capturing of Opera It follows issues concerned with specific recording scenarios which relate to the points above: 2.2.1. Capturing of the whole opera: It will almost always be preferred that the whole opera is digitized in greater or lesser quality on video and audio. The complexity of the recording will depend on content and the costs involved. Smaller parts of the opera (possibly single titles, arias, etc) could be captured in a much more complex form so that users * can fade certain instruments or singers down * can "mix" balance diffe rent instrument groups/singers The quality of the recording depends on a large part on the costs involved. Recording Scenarios in short can be any of the below but require tight and good planning * Single-Session, Straight to stereo recording at the Opera House including video * Single Session, Orchestra and Singers "straight to multitrack" recording at the Opera House incl. video * Multi-Session, Orchestra multitrack recording with separate recording session for singers and video 340

Page  00000341 * Multi-Session, Multitrack recording of instrument groups and singers in Studio, Video only at Opera House * Multi-Session, Multitrack recording of instrument groups and singers at Opera House, Video only at Opera House 2.2.2. Capturing artefacts of the production proc ess while being in rehearsal / planning / production stage: It is envisaged that there will be opera objects recorded while the opera is "in the making". This can include pictures of drafts of set and costumes, as well as pictures or audio/video recordings of team meetings and f~irst rehearsals. In the simplest form this means that an individual present in the opera would take shots with a digital cameras, or the scanning in of drafts and sketches. In a more complex form small camera teams will need to be planned into the production phases (sound technician and camera man), recording separate Opera "events". The decision of what is being captured and when will need to depend solely on the creative input of the producer/director of the production and the available budget. 2.2.3. DVD issues: The web-based portions of the OpenDrama project will be combined with internet enabled/interactive DVD format disc. As well as openingup new technology based revenue streams, existing opera attendees and pro-active supporters of opera would be brought closer together and foster a greater sense of community. Because the technology being utilised allows for nonlinear navigation and multi-channel data (jumping from aria to aria, isolation of particular parts of the orchestra, dynamic libretto representation and running 'textual commentary') it will allow for a substantial amount of information to be offered to a user. With the use of enhanced-mode DVD's (also re ferred to as WebDVD's) a mixture of traditional DVD data (MPEG2 video/audio) could be combined with the ability to take advantage of a PC's DVD-ROM drive and internet connection to connect to a server and provide additional content and features not available at the time of DVD mastering. The WebDVD's could contain pre-recorded, highquality, material such as detailed background of the plot, information on the principal actors, live footage of dressrehearsals and combine this with actual broadcasts of the real performance via the OpenDrama service. The combination of 'must-have' background information together with the tasters and teasers of the live performance the DVD will be a sought-after item of the opera-going public. The value of the DVD is increased, both in it's revenue making potential for opera Houses in general and it's source of ever-changing content for the user by combining it with the interactive elements. These additional on-line services may be offered to a user on a subscription, or one-off micropayment, basis. These DVD could also target the notion of generating traffic volume on the networked opera website, where latest shows/offers/promo's would receive a large number of eyeballs or would it be seen as a value-added feature. As an alternative a traditional DVD, without webinteractive content could be produced which could contain elements of the above and below discussed. 2.2.4. Community: A closer sense of community between Opera Houses and it's patrons could be created by using the interactive services to, as an idea, allow internet users to chat to the producer/principals after a performance in a chat-room for an hour after a performance. Discussions and web-based presentations could be given using the networked opera services, with archives of past sessions being made available (either as free or for a fee). This could be similar to what current Opera Houses are rm mentarily providing as services for its "Friends", i.e. talks and seminars and study days. These talks could be digitised and made available to provide more insight into the production process and, increasing their feeling of ''ownership'' and ''where their money was actually spent''. These services could possibly increase the attractiveness of being a "friend", which is preferred by Opera as well as "friends" in order to maintain a high quality service with a stable budget. Additional material has the potential to take a viewer/user into the mind of the shows producer and provide an insight into the creative journey they normally only see the end result of. Part of this additional material could be a 'DocuOpera', a separate 'making-of documentary covering the whole of the pre -production of the show, from the producers or actors/singers perspective. 3. Conclusions The OpenDrama Project is currently doing its first steps with using two legs simultaneously: it will try to integrate the "human-resource heavy" world of opera production logistics with the "code-heavy" world of software integration management, hopefu~lly resulting in an inherently musical and operatic experience which will necessarily include the "processing-heavy" world of streaming structured data. And having done all that, it will hopefully found and proven along the way the production methodologies needed to join both together in order to network opera and its digital cultural heritage for a future generations of opera interested communities. http://www. opendrama. com 341