Page  00000279 DIGITAL PRESERVATION OF INTERACTIVE MULTIMEDIA PERFORMANCES Kia Ng, Tran Vu Pham, Bee Ong, Jerme Barthelemy, 2 Alain Bonardi, 2 David Giaretta3 SICSRiM - University of Leeds, School of Computing & School of Music, Leeds LS2 9JT, UK kia@icsrim.org.uk www.icsrim.org.uk ABSTRACT Interactive multimedia technologies and digital media are now usual tools and resources for contemporary performing arts. From the perspective of recreating a performance and/or later analysis, these technologies and media have added much complexity to preservation issues. Preserving an interactive multimedia performance involves keeping human interactions with multimedia systems, the multimedia contents generated (usually music and graphical animation) and the artistic, cultural and technical knowledge embedded in the performance material so that a recreation of the performance is possible in the future. This paper introduces an approach to such a challenging preservation by using 3D motion data for coding human interactions and ontologies for knowledge preservation together with the preservation framework developed in the CASPAR EC IST project, which is based on the standardised Open Archival Information System (OAIS) reference model. The work reported is part of the contemporary arts testbed of CASPAR. 1. INTRODUCTION Interactive multimedia technologies are popularly used in contemporary performing arts, including musical compositions, installation arts, dance, etc. Typically, an interactive performance may involve one or more performers who interact with a computer based multimedia system making use of multimedia contents that may be prepared as well as generated in real-time including music, manipulated sound, animation, video, graphics, etc. with various electronic devices, computer software and hardware. The interactions between the performer(s) and the multimedia system can be inferred in a wide range of different approaches, such as body motions [7, 9], movements of traditional musical instruments, sounds generated by these instruments [10, 11], tension of body muscle using bio-feedback [8], heart beats, sensors systems, and many others. These "signals" from performers are captured and processed by the multimedia systems. Depending on specific performances, the "signals" will be mapped to multimedia contents for generation using a mapping strategy. Figure 1 illustrates a typical production process involved a performance using MvM (Music via Motion) interactive multimedia system [9]. In the MvM the 2 IRCAM 1, place Igor-Stravinksy, 75004 Paris, France 3 STFC, Rutherford Appleton Laboratory, Space Data, Space Science and Technology Department, Oxfordshire OX11 OQX, UK signals are movements of the performer captured in 3D using a motion capture system. The motions carried in the 3D motion data are then analysed and mapped into multimedia contents using a predefined mapping strategy. Interactive multimedia performances are usually ad hoc. Manipulating multimedia contents using computers is an essential part of a live performance. Using only performance outputs recorded as audio and video medias will not be sufficient for a proper analysis (e.g. for studying the effect of a particular performing gesture on the overall quality of the performance) or recreation of a performance at a later time. In this context, traditional music notation as an abstract representation of a performance is also not capable of storing all the information and data required to recreate the performance. Therefore, in order to keep a performance alive through time, the whole production process to create the output needs to be preserved. Preserving the whole production process of an interactive multimedia performance is a challenging issue. In addition to the output multimedia contents, related digital contents such as mapping strategies, processing software and intermediate data created during the production process (e.g. data translated from "signals" captured) have to be preserved, together with all the configuration, setting of the software, changes (and time), etc. The most challenging problem is to preserve the knowledge about the logical and temporal relationships amongst individual components so that they can properly assembled into a performance during the recreation process..........:..I................. T. Figure 1. MvM multimedia performance production process CASPAR - Cultural, Artistic and Scientific knowledge for Preservation, Access and Retrieval - is addressing such challenges in digital preservation by employing OAIS reference model [2] to develop a 279

Page  00000280 framework for the preservation of digital assets from various domains [1]. It is co-supported by the European Commission, under the Information Society Technologies (IST) Sixth Framework Programme and being carried out by the consortium consisting of 17 research institutions and laboratories from academic and industrial sectors across Europe. In this paper, the focus is particularly on the use of 3D motion data for digitalising motions of performer(s) and ontology to address the issues related to preservation of interactive multimedia performances within the CASPAR framework to keep knowledge about performances alive through time. The next section of this paper discusses the underlying CASPAR preservation framework. In Section 3, the focus is on the rationale for using 3D motion data for preservation and the techniques and tools for capturing the motions. Section 4 explains the use of ontology for preservation of knowledge about an interactive performance and the current work on development of ontology for interactive multimedia performance domain. The paper is concluded by the current status of this initial work and its future direction. 2. CASPAR PRESERVATION FRAMEWORK CASPAR aims to build a framework to support end-toend preservation lifecycle for scientific, artistic and cultural information. The main objectives of CASPAR include: * establishing a foundation methodology applicable to an extensive range of preservation issues; * researching, developing and integrating advanced components to be used in preservation activities. These components will be the building blocks of the CASPAR framework; * creating the CASPAR framework: the software platform that enables thebuilding of services and applications that can be adapted to multiple areas. Three testbeds are being developed within CASPAR to instantiate its generic framework functionalities into real domains: (i) a scientific testbed for very-high volume, complex digital data objects, oriented towards processing; (ii) a contemporary arts testbed dealing with dynamic interactive digital objects, oriented towards presentation and replay of artistic contents; (iii) a cultural data testbeds for virtual digital objects, spanning between processing and display. The focus of CASPAR is specifically on preserving knowledge for future archive intelligibility and information system/services interoperability. Preserving information and knowledge - not just 'the bits' - allows the keeping of archives alive through time. Therefore, in addition to simple data syntactic of data objects, CASPAR will also capture their higher-level semantics for preservation. CASPAR framework is based on OAIS, which is an ISO standard for archival information systems [2]. OAIS defines a consistent set of terminologies, conceptual and functional models for the development of archival information systems. The adoption of OAIS in CAPSAR will enable its interoperability with other OAIS based digital archives. CASPAR adds to OAIS a number of high-level components to deal with a range of issues involved in the digital preservation process, such as knowledge preservation, virtualisation of multidisciplinary contents, preservation registry and storage virtualisation, as shown in Figure 2. As a preservation system, the components of the CASPAR must themselves be preservable. 3. PRESERVATION OF 3D MOTION 3D motion data plays an important role in interactive multimedia performances particularly when motions/gestures of performers are used as input, for later analyses/recreation of the performances. It may not be easy even for the same performer to expresses precisely the same motion/gesture at a later time. Therefore, in cases, where the original performer is absent, preservation of the performer's motions during a performance is really necessary for a proper analysis of the performance at a later time. In these cases, 2D video images are not sufficient, as they only show the views of the motions from particular angles. For completeness, the motions need to be captured and visualised in 3D. The MvM interactive multimedia performance system shown in Figure 1 is using this approach. The system can be used for a live interactive performance as well as for analysis of musicians' playing traditional instruments such as a violin, cello or viola. Post performance analyses can be on particular body gestures of the performer or the relative movements between instruments. For example, when playing a violin, the movement of the bow relatively to the instrument can be helpful for analysing the skills and the expression of the performer. In terms of preservation, the difficulty raised by using 3D motion data is enabling the interoperability amongst different data formats and applications. There are currently many different systems using different technologies for capturing and processing 3D data, such as the Polhemus magnetic systems which use magnetic sensors positioned in a magnetic field for tracking movement or the Vicon optical systems using a set of infrared cameras for tracking reflective markers which are attached to the captured subjects. Each of these systems may support a number of different data formats. The common 3D formats exist in the community are usually marker-based, e.g. Coordinate 3D (C3D) and Tracked Row Character (TRC), or skeleton-based, e.g. BioVision Hierarchical Data (BVH) and Hierarchical Translations and Rotations (HTR). Some application specific formats, such as the Vicon V-file and Trial file, contain both marker and skeleton data in one file. The conversion of data from one format to another may 280

Page  00000281 Figure 2. CASPAR preservation framework result in loss of some essential data (e.g. converting data from a skeleton-based format to a purely marker-based format usually results in a loss of segment orientation data). The Gesture Description Interchange Format (GDIF) has been proposed as an attempt to provide a common interoperable format for describing music related gesture [6]. Currently, within CASPAR framework, any data format used is required to be accompanied by documentation about its data structure, known as representation information. This helps to interpret the data in the archive and enables all kind of representations to be included and packaged for preservation. 4. ONTOLOGY FOR INTERACTIVE MULTIMEDIA PERFORMANCES The use of ontology supports the preservation process from a different perspective and at a higher level. It helps to preserve the knowledge about the relationships amongst different elements of an interactive multimedia performance so that a successful recreation of the performance at some time in the future by assembling preserved elements together is possible. 4.1. Elements of an Interactive Multimedia Performance A typical interactive multimedia performance (e.g. the MvM) involves hardware equipments, software applications and people (e.g. performers, directors, and sound engineers). Although in digital preservation, only digital objects are preserved in digital archives, it is also necessary to document, on one hand, the relationships between digital objects and their external context, and on the other hand, the embedded artistic and technical content. This knowledge is essential to a performance. Through time, elements that are used in the original performance may be missing at the time of recreation. The preserved knowledge can help to find alternative equipments to replace the missing ones. For example, in a performance using MvM system, the following elements and their inter-relationships need to be documented: * Digital data objects, e.g. 3D data, video, audio, 2D images * Software applications, e.g. Max/MSP patches, PD patches, etc * Hardware equipments, e.g. Vicon 3D capture system, microphones, video cameras and computer systems, synthesisers * Musical instruments, e.g. violins, cellos or violas 281

Page  00000282 * People, e.g. the director, performers, MAX/MSP programmers and sound engineers In addition to documenting the elements and their relationships, the different stages of a performance production process, in which the elements are involved, and their temporal relationships also need to be documented. 4.2. Extending CIDOC-CRM and FRBRoo The CIDOC Conceptual Reference Model (CRM) is being proposed as a standard ontology for enabling interoperability amongst digital libraries [3]. An advantage of CIDOC-CRM is that it has a basic set of well defined concepts for physical as well as temporal entities. This is important for describing temporal dependencies amongst different elements of an interactive multimedia performance. Together with FRBRoo [4], an object oriented version of FRBR (Functional Requirements for Bibliographic Records) [5], CIDOC-CRM can possibly be used to describe an interactive multimedia performance at a high level. However, its current version lacks vocabulary for describing digital objects involved in an interactive performance in the digital preservation context. New vocabulary has been introduced to further specialise the generic concepts of CIDOC-CRM and FRBRoo for describing objects specific to interactive multimedia performance domain. For example, in order to describe a performance and involved elements a number of concepts such as 'Performance', 'Performance Activity', 'Instrument', 'Equipment' and 'Digital Object' together with their associated relations have been introduced. The concept "Digital Object" has also been further specified with sub-concepts for documenting the relations amongst digital objects, as shown in Figure 3. Digital Object Digital Data Container Digital Data Object Computer Program Operating System Software Application Figure 3. Specification of digital objects 5. CONCLUSION This paper has presented characteristics and requirements for preservation of interactive multimedia performances. The OAIS-based CASPAR framework has been introduced as a solution to a wide range of digital preservation, which also addresses the preservation of interactive multimedia performances in one of its testbeds. This paper focused on 3D motion data and proposed the application of ontology. It is now important to gain involvement from related communities to standardise the ontology for describing the interactive multimedia performances for the preservation process. 6. ACKNOWLEDGEMENT Work partially supported by European Community under the Information Society Technologies (IST) programme of the 6th FP for RTD - project CASPAR contract IST-033572. The authors are solely responsible for the content of this paper. It does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of data appearing therein. 7. REFERENCES [1] CASPAR, "Cultural, Artistic and Scientific knowledge for Preservation, Access and Retrieval", http://www.casparpreserves.eu/. [2] Consultative Committee for Space Data Systems, "Reference Model for An Open Archival Information System", 2002, http:i//public.ccsds.org/publications/archive/65x0bl.pdf. [3] Doerr, M., "Increasing the Power of Semantic Interoperability for the European Library", in ERCIM News, vol. 66, 2006. [4] Doerr, M. and LeBoeuf, P., "FRBRoo Introduction", 2006, http://cidoc.ics.forth.gr/frbr inro.html. [5] FRBR, "Functional Requirements for Bibliographic Records - Final Report", International Federation of Library Associations and Institutions (IFLA), Frankfurt am Main, Germany 1997, http://www.ifla.ore/VII/s13/frbfr/rbr.pdf. [6] Jensenius, A. R., Kvifte, T., and Godoy, R. I., "Towards a gesture description interchange format", Proceedings of New Interfaces for Musical Expression - NIME 06, IRCAM - Centre Pompidou, Paris, France, 2006. [7] Morales-Manzanares, R., Morales, E. F., Dannenberg, R. B., and Berger, J., "SICIB: An Interactive Music Composition System Using Body Movements", Computer Music Journal, vol. 25, pp. 25-36, 2001. [8] Nagashima, Y., "Bio-Sensing Systems and Bio-Feedback Systems for Interactive Media Arts", Proceedings of 2003 Conference on New Interfaces for Musical Expression (NIME-03), Montreal, Canada, 2003. [9] Ng, K. C., "Music via Motion: Transdomain Mapping of Motion and Sound for Interactive Performances", Proceedings of the IEEE, vol. 92, 2004. [10] Overholt, D., "The Overtone Violin", Proceedings of International Conference on New Interfaces for Musical Expression, Vancouver, BC, Canada, 2005. [11] Young, D., Nunn, P., and Vassiliev, A., "Composing for Hyperbow: A Collaboration between MIT and the Royal Academy of Music", Proceedings of International Conference on New Interfaces for Musical Expression, Paris, France, 2006. 282