Page  00000077 ARTABILITATION ICMC PANEL PAPER DENMARK 2007: NON-FORMAL REHABILITATION VIA IMMERSIVE INTERACTIVE MUSIC ENVIRONMENTS Anthony Brooks, Aalborg University Esbjerg, Denmark; Barry Eaglestone, University of Sheffield, UK; Phil Ellis, University of Sunderland, UK; Rolf Gehlhaar, Coventry University, UK; Luis Miguel Girdo, Artshare, Aveiro, Portugal and Coventry University, UK; Wendy Magee, Institute of Neuropalliative Rehabilitation, Royal Hospital for Neuro-disability, UK; Eduardo Miranda, University of Plymouth, UK; Eva Petersson, Aalborg University Esbjerg, Denmark; Paulo Rodrigues, Casa Da Musica, Porto, Portugal Contact author: tonybrooks@aaue.dk ABSTRACT This paper brings together perspectives of the ICMC 2007 ArtAbilitation Panel on non-formal rehabilitation via immersive interactive music environments. Issues covered are sound therapy and music therapy, musical topologies, brainwave control and research methodology. 1. INTRODUCTION Enabling interactive music-making for people with complex needs provides opportunities for them to knowingly become active agents in musical dialogues. Music technologies, sensitive to tiny or inconsistent gestures, enable self-expression, heighten self-awareness in relation to others and contribute to identity development on many levels [12]. Thus, music technologies are understood as a motivational tool in wider rehabilitation and care of people with complex needs. Also, these tools have the advantage of being used independently by people otherwise heavily dependent, thus empowering and enhancing feelings of achievement. This paper brings together insights of the ICMC 2007 ArtAbilitation panel, to provoke discussion of the embodied aesthetic and related issues. Section 2 motivates this area of research from sound therapy [7] and music therapy perspectives. Sections 3 and 4 respectively focus on design of the 'musical instrument' and the means of 'playing' it. Section 5 presents an information science perspective on further research. Concluding remarks are given in section 6. 2. A SOUND THERAPY AND MUSIC THERAPY PERSPECTIVE New technology has increasing importance for people with disabilities. Designing interactive music environments for inclusion, involvement and enjoyment implies interactive experiences, which concern active participation in activities, leading to knowledge or skill [16]. These experiences encourage engagement in the activity out of self-interest and curiosity (intrinsic) rather than an activity introduced by another (extrinsic). Thus, design configures learning resources and interaction [10]. Aesthetic resonance [5] [6] has a clear focus on participants in their making and transformation of meaning through action cycles. Designing for aesthetic resonance emphasises action cycles as intertwined aspects of non-formal rehabilitation processes. In doing so, we transcend mere use of interactive music environments towards exploration and transformation, thus considering action cycles as new creations [2]. Patterns of action that emerge through exercise become constituents for new patterns of action directed at more complex tasks [2]. Exploration goes with play, but is not the same. [3] describes how play involving manipulation of tools, requires competence achieved through a learning process starting with exploration of characteristics of the tools. Through action cycles, participants experience and play with sensation offered by interactive attributes. This exploration-to-play process facilitates discovery of interesting and surprising content. Therefore, the interface must be flexible enough to facilitate participants' unanticipated desires. Absence of negative consequences encourages exploration, which in turn, can result in a sense of involvement (immersion) and development of unemployed skill. By this, the focus is on the attributes of the interactive music environment and the rehabilitation process/outcome [14]. This offers a new approach to rehabilitation by emphasising participant's creation of meaning and production of expressions. This approach does not take aspects of the rehabilitation process and outcome for granted, neither being coerced, but rather strategised into play and creative activities that are inherent to e.g. art making [1] [9]. By this, play and creativity at the participant level conceal embedded training and learning available from the designed interactions with the feedback media. Thus, learning is subliminal for a user engaged in the responsive environment. Motivation is suggested as optimised through action cycles where the user iteratively explores and transforms the feedback media. This process contains choices and decisions that indicate learning, e.g. as increased repertoire of expressions, changes of skills, new patterns of social interaction. Action cycles, comprising iterative loops of exploring and transforming, constitute part of a theoretical map for analysing critical incidents in a non-formal rehabilitation process. These cycles are related to participant's learning experience. The other part of the theoretical map concerns design issues in the form of 77

Page  00000078 use qualities relative to the participant's interactive experience [2]; transparency, social-action space, user control/ autonomy, pliability, playability and seductivity [11]. Interaction should be designed to enhance a participant's control and, thereby facilitating interest and motivation. Action cycles inherent in the interaction create a process of enticement by attracting the participant's attention, ability to make progress and experiencing fulfilment by ending the experience in a positive way. Thus, the person is seduced by the system's playability offering surprise and prompting emotional responses through interactional beauty [11] of auditory qualities. 3. MUSICAL TOPOLOGY OF THE 'INSTRUMENT' Realisation of therapeutic benefits outlined above requires innovative installations with potential for meaningful music making within constraints imposed by the music maker's disabilities. Critical issues are respectively discussed in this and section 4, i.e., the design of the 'keyboard' of an 'instrument' and the means of 'playing' it. Here we discuss the concept musical topology, as has emerged through the work of Rolf Gehlhaar on his SOUND=SPACE installation [8]. A musical topology results from analysis and processing of information gathered from movement of bodies in a space equipped with sensors. The information is fed as control variables to compositional algorithms, and via synthesis routines, produces sounds. Thus, the audience becomes the performers. These topologies are passive, active, or hybrid. An example passive topography is simple triggering of a sound(s) with specified duration(s) by a person stepping into the area to which the sound(s) is assigned; like walking on imaginary keyboards that span the space. The 'instrument' is deterministic, playing only when someone triggers it, sounds 'mapped' onto the space by the program. Each 'keyboard' can be structured independently or assigned to a group of keyboards in a specific region, each with different pitches, durations and sounds assigned to the 'keys'. An active topology comprises an algorithmic realtime composition by the computer, influenced by presence and movement of persons in the space. Movements are converted into control parameters of the composing algorithm. The effect is like conducting an ensemble of musicians: usually, greater activity results in more animated and complex music. The algorithm employs interlinked chains of probability matrices, which can be programmed to produce generic musical styles. The hybrid topology combines both of the above into a space that reacts not only to movement but also to position. The effect can be like controlling tempo or direction of a musical flow by moving about the space and, at the same time, triggering specific events by stepping into specific places. SOUND=SPACE implements the above topologies as a musical 'instrument' 'played', usually by several persons in an empty space surveyed by an ultrasonic echolocation system that detects positions and movement of people. These measurements are sent to a computer that converts them into sounds. Normally, the space is square (up to 10m x 10m) sufficient for 8-15 people. Sensors on two contiguous sides look inwards across the space, creating a 'grid'. Thus, SOUND=SPACE is a complex, sophisticated multifunctional, multi-user system; but its environment is uncomplicated, friendly and non-intimidating. When invaded, it responds immediately with sound. No expertise is required to create generally exciting, engaging and pleasant sounds and musical sequences. Since moving in space is key to playing this musical instrument, users improve their perception of space, and, consequently their capabilities of movement. Accuracy only becomes a requirement when an intentional (musical) gesture is desired. Thus, SOUND=SPACE allows musical expression with no special skills while promoting development and improvement of new skills. Potential for benefiting people with disabilities emerged during an installation at the Gulbenkian Foundation, Lisbon in 1986. A visit by disabled children from special schools was such a success that it catalysed Gehlhaar's intensive involvement with SOUND=SPACE in the world of disability. He came to understand that technology no longer allows us simply to make art for a public; it demands that we create opportunities for the public to make art. It is about creating situations that encourage active, creative response common to all humans, able or disabled. Accordingly, he redesigned SOUND-SPACE to tailor the interactive aspects more towards the highly varied skill sets of disabled users. A further dimension emerged during recent SOUND=SPACE workshops run by Luis Miguel Girao in Oporto, Portugal, as part of "Ao Alcance de Todos". The system was set-up for participants suffering from brain paralysis, educators and music students, in a public space that also allowed interaction with passersby. This highlighted SOUND=SPACE as a social environment also including, perhaps more importantly, interaction between person and person (disabled or nondisabled) while 'playing' in the environment. Thus, it was evident that non-disabled people had much to learn from disabled people about awareness and sensitivity to the musical environment. Finally, we note that environments, such as SOUND=SPACE, have an educational role. Techniques of progressive exploration of the instrument reveal basic principles of playing music, such as: sound/silence, musical phrase and musical dialog. A deeper approach brings up the learning of harmony and rhythm. Gehlhaar has developed different versions of the topographies, each with its own characteristic mood and nature - calming, exciting, sustained, rhythmical, percussive, and so forth. It is possible, within a workshop, to move rapidly from one mood to another, with few words spoken, to encourage participation, to support the mood and activity of the moment, to focus attention or to shift the concentration of the participants, to calm them down when they get too excited. 78

Page  00000079 4. THE MEANS OF EXPRESSION: MIND CONTROL AND THE BCMI-PIANO Use of movement-sensor based systems, such as SOUND=SPACE, is limited by disabilities which severely impede movements. This section reviews work towards an exciting alternative, i.e., 'mind control'. The BCMI-Piano takes brain activity as input to control music making. To this end, the system comprises 4 main modules: braincap, analysis, music engine and performance. Braincap generates the electroence-phalogram (EEG) from electrodes on the scalp of the performer, via a biosignal amplifier and a real-time acquisition system. The analysis module performs EEG analyses in real-time to generate two streams of control parameters: (i) information about the most prominent EEG frequency band, extracted using power spectrum analysis; (ii) information about complexity of the signal, extracted using Hjorth analysis. The first stream is used by the music engine, to generate the music (applying a set of generative music rules, each of which produce a musical bar, or measure). Currently two styles of music are activated, depending on the EEG's salient frequency level. The second stream controls tempo of the music. Every time the music engine has to produce a bar, it checks the EEG power spectrum and activates rules associated with the prominent EEG rhythm in the signal. The system is initialised with a reference tempo, which is constantly modulated by the signal complexity analysis. MIDI information is output for performance, currently using the Yamaha Disklavier piano. Miranda et al also are investigating methods to train subjects to achieve specific EEG patterns to play the BCMI-Piano system, to achieve greater musical control. Evidence suggests this is possible using biofeedback. In addition, they are seeking improved understanding of EEG analytical semantics in relation to musical cognition. [13], and to address the non-ergonomic nature of the electrode technology for sensing the EEG, which can be awkward to wear and uncomfortable. This research presents possibilities for recreational devices for people with disabilities and instruments for concert performance and composition. The BCMI-Piano provides opportunity for those physically unable to play the piano in the traditional sense to do so using EEG signals, hence empowering themselves and possibility establishing themselves as musicians. Also, many music-making devices work effectively for people with disabilities, but often allow insufficient control for the severely physically disabled. Training the latter to control BCMI technology to compose and perform music could mean greater control and independence for musicians and aspiring musicians with disabilities. 5. FURTHER RESEARCH: AN INFORMATION SCIENCE PERSPECTIVE Research in this area is inherently multi- and interdisciplinary. Primary issues are medical and psychological, since physical and mental well being of the users must be paramount. Also necessary is the inventor who empathises with the users and their needs, and has creativity and technological competence to invent appropriate configurations. Composers and musicians are required to provide musical and creative insights and drive aesthetic and artistic aspects of the research. Design and engineering of installations requires IT and computer science expertise. Also, training is required to ensure adequate skills for using these tools effectively with people who have complex needs [12]. To the above, the information scientist should be added, to provide methodology to understand, evaluate and optimise such systems so as to maximise their efficacy. However, the information science contribution is mainly missing. An information science perspective derives from parallels between research issues relating to the above systems for people with disabilities and composition software systems [4]. An open research question for both is, how can systems be engineered such that they best meet creative aspirations of their users? Consequently, it is necessary to determine cognitive processes of the subjects, which is intrinsically difficult. As Laske (cited by [15]4:31]) observed: "the kind of musical knowledge that, if implemented, would improve computer music tools is often not public or even shared among experts, but personal, idiosyncratic knowledge...the elicitation of personal knowledge and of action knowledge still awaits a methodology...." Therefore empirical studies are necessary to reveal what actually happens during the composition process. Arguably, this requires qualitative inductive research methodology, involving naturalistic study, in preference to quantitative methods which attempt to 'measure and generalise'. Specifically, research methodology that has proved effective in identifying tensions between composers and composition software systems [4] may prove effective. These methods involved naturalistic study of a small number of professional composers working on their own compositions using the hardware/software system with which they are familiar. Rich data was collected by a range of means, including pre- and post-session interviews, verbal scene-setting, and, during composition sessions, videotaping, interrogative observation, and encouraging think-aloud and reflection. Data was analysed in a non-linear way by coding sections of all data types produced and placing them into categories. Relationships between categories were then established as the basis from which models of the compositional process were derived. Thus, the spirit of a qualitative grounded theory approach was applied, as expounded in [5] (also used in [12]) whereby "The model derived should organise the features or the data in a coherent form that relates both to the perceptions and concepts of those studied and to the viewpoint that the researcher is developing. In that sense, although the concepts are derived from the data, they are not simply a restatement of the data. In developing the model with its attendant categories, properties, and relations, the researcher embodies the perceptions and activities of those studied in the model but in a way that allows them to be understood in other terms." In the spirit of qualitative research, theories inductively emerged from 79

Page  00000080 the data to explain phenomena within the context of the rich situations within which they occurred. The above approach potentially can be adapted and effectively used to also identify, for example, tensions that may exist between aspirations of disabled users of the types of systems discussed here, and the means provided for them to express themselves. In particular, collecting and aligning multiple streams of observation data in various media allows focus on the richest and most informative data sources, while seeking confirmation and reinforcement of analysis from other less rich sources. 6. CONCLUDING REMARK Multi-sensory interfaces enabling expressive dialogue for people with mental and physical impairments often results in higher levels of self-awareness, interaction and control. It also results in increased insights about a participant's motivation to express, revealing more of the person that is often hidden behind the salient impairments. More research and development is needed to support control in optimal ways to allow for development in e.g. type and complexity of interactions. Individually experienced complementarities between aspects of the environment to be controlled (e.g. timbre, volume, colour etc) and the physical movement executing the control (eye, head, hand movements etc.) are not fully understood. Better understanding of these will provide design requirements for a new generation of adaptive interfaces supporting developmental processes in expression and communication. 8. REFERENCES [1] Bloom, A. (trans.ed.). The Republic of Plato. New York. Basic Books, 1968. [2] Bruner, J.S. "Organization of Early Skilled Action", Child Development, 44(1), 1-11, 1973. [3] Bruner, J.S. "Nature and Uses of Immaturity", 1972. In Bruner, J.S, Jolly, A., Sylva, K. (Eds.) Play - Its Role in Development and Evolution. United States: Basic Books, 1976. [4] Eaglestone B., et al. "Information systems and creativity: an empirical study", Journal of Documentation, 63(4), 2007. [5] Ellis, D. "Modelling the information seeking patterns of academic researchers: A grounded theory approach", Library Quarterly, 63(4): 469-486, 1993. [6] Ellis,P. "The Music of Sound: a new approach for children with severe and profound and multiple learning difficulties", British Journal of Music Education, 14:2, 173-186 CUP, 1997. [7] Ellis,P. "Moving Sound". In Gallagher, MacLachlan (Eds), Enabling Tech nologies. Body, Image and Function, Churchill Livingstone, 2004. [8] Gehlhaar, R. "SOUND=SPACE" Contemporary Music Review 6(1): 59-72, 1991 and Sparacino, F., Davenport G., Pentland A. "Media in Performance: Interactive spaces for dance, theatre, and museum exhibits", IBM Systems Journal, 39(3-4): 479-510, 2000 [9] Krentz, A.A. "20th WCP: Play and Education in Plato's Republic", 1998. In: Steiner, D. (ed.) Twentieth World Congress of Philosophy, Boston, Massachusetts. http://www.bu.edu/wcp /Papers/ Educ/EducKren.htm [10] Kress, G., van Leeuwen, T. Multimodal Discourse. The modes and media of contemporary communication. London: Arnold, 2001. [11] L6wgren, J., Stolterman, E. Thoughtful Interaction Design. A Design Perspective on Information Technology. Cambridge, Massachusetts: MIT Press, 2004. [12]Magee, W.L., Burland, K. Exploring the use of electronic music technologies in clinical music therapy. Establishing definitions and scope of practice. Final report. London: Royal Hospital for Neurodisability, 2006. [13]Miranda, E.R., et al "On Harnessing the Electroencephalogram for the Musical Braincap", Computer Music Journal, 27(2):80-102, 2003. [14] Petersson, E. "Non-formal Learning through Ludic Engagement within Interactive Environments", Dr. Scient. Thesis, Lund University, School of Teacher Education, Malmo University, 2006, http://hdl.handle.net/2043/2963 [15] Polfreman, R. "A task analysis of musical composition and its application to the development of Modalyser", Organised Sound, 4 (1), 1999. [16] Rogoff, B. Apprenticeship in Thinking. Cognitive Development in Social Context. New York: Oxford University Press, 1990. 80