Page  00000045 Sound Synthesis Affected by Physical Gestures in Real-time Lars Graugaard Department of Software and Media Technology Aalborg University Esbjerg lag@cs.aue.dk Abstract Motivation and strategies for affecting electronic music through physical gestures are presented and discussed. Two implementations are presented and experience with their use in performance is reported. A concept of sound shaping and sound colouring that connects an instrumental performer's playing and gesturest to sound synthesis is used. The results and future possibilities are discussed. 1 Introduction Musical expression has strong ties with physical gestures for different purposes and in different contexts (Camurri et al. 2005, Dahl and Friberg 2004). This demonstration presents motivations and strategies for including a performer's gestures as part of the music performance, at times as the primary expression link between performer and music. 2 Case 1: Gestures You Made My composition Gestures You Made for oboe and interactive computer is mainly interactive through sound analysis (Graugaard 2005/06). But towards the end of the piece the player stops playing at moments, and instead affects the sound synthesis by assuming body positions and making full-body gestures inside a 'videospace' adjacent the performance space. The link between the two affect channels is the parallelism between expression in sound and in human gesture. The parallelism is underscored by the fact that the two channels affect the same sound, which at times is done simultaneously through playing and physical gesture. It is straight-forward that playing in interactive music can produce and affect electronic sound, but it is interesting to note that the production and affect of sound through physical gestures is just as readily accepted as a valid connection by the audience. 3 Case 2: Stgo Laptop performance has developed its own performance practise of an apparently very unactive performer participation, bordering on the enigmatic (Cascone 2000). Informing laptop music with body gestures may take laptop performance out of this attitude towards the concert situation - or it may make it even more enigmatic and ritualistic - but the artistic reason to include video tracking in Stgo (Graugaard 2005) was to instill human expressiveness into procedural electronic music in real time. The sensation of shaping the sound is strong, certainly because the movements and the resulting sound is easily related, but also very much because the hands are our primary shaping tool. Likewise, the hands shaping the sound is readily perceived by the audience, even though not all its relation is directly traceable. Choises of tracking and mapping strategies became important since no sound input is present. 4 Conclusion Informing sound synthesis with human gestures in real time is a very potent performance enhancement technique. It adds a dimension to the performance where the visual presence of the performer extends into the sound in a direct and unambiguous way, and enhances the music's presence to envelop a physical presence. This is in effect a multimodal interactive music system because an informative channel is added to the synthesis process or high-level proceedings that isn't easily established through other means. References Camurri, A., Volpe, G., De Poli, G., Leman, M. (2005): Communicating Expressiveness and Affect in Multimodal Interactive Systems. In IEEE Magazine Jan-March pp. 43-53 Cascone, K. (2000): The Aesthetics of Failure: "Post-Digital" Tendencies in Contemporary Computer Music. In Computer Music Journal, 24:4, pp. 12-18, Winter 2000. Massachusetts Institute of Technology. Dahl, S. and Friberg, A. (2004): Expressiveness of musician's body movements in performances on marimba. In A. Camurri, G. Volpe (eds.) Gesture based Communication in Human - Computer Interaction, LNAI 2915 (pp.479-486), Berlin, Springer Verlag. Graugaard, L. (2005): Stgo. Premiered December 6th 2005 by Lars Graugaard; Cech Santiago (Chile). Graugaard, L. (2005/06): Gestures You Made. Commissioned and premiered February 12th 2006 by Eydis Franzdottfr; Dark Music Days 2006, Reykjavik (Iceland). 45