Page  00000485 TRANSLATING DANCE MOVEMENT INTO MUSICAL RHYTHM IN REAL TIME: NEW POSSIBILITIES FOR COMPUTER-MEDIATED COLLABORATION IN INTERACTIVE DANCE PERFORMANCE Carlos Guedes ESMAE-IPP, Porto ESART-IPCB, Castelo Branco Portugal ABSTRACT In this paper, I describe the theoretical background that led to the creation of software for interactive dance performance that interprets rhythm in dance movement as musical rhythm. This software was implemented as a library of external objects for Max/MSP[12] that processes data from an object or library that performs frame-differencing analysis of a video stream in real time in this programming environment. After describing the main aspects that establish the relationship between movement in dance and musical rhythm, I explain how the software implements this knowledge, and briefly describe the creative work done so far utilizing this software. 1. INTRODUCTION Improving interactive computer systems in music and dance performance by borrowing knowledge from several different research areas, namely human perception and cognition or music and movement theory, has proven to be a fruitful endeavor in the past 15 years or so. The work by Robert Rowe [19][20] in interactive music systems and the work by Antonio Camurri and colleagues in the development of EyesWeb's [14] movement analysis modules stand out as two good examples of how can one apply such research towards the development of more elaborate modes of interaction between humans and computers in interactive performance. In this paper, I discuss the theoretical background that led to the creation of a software library for Max/MSP that extracts musical cues at the temporal level from dance movement that borrows existing knowledge from the fields of music theory and cognition and dance and movement theory. This software library allows a dancer to control the tempo of an electronically-generated music score, and/or to generate musical rhythmic structures from bodily movement in real time during a performance. 1.1. How Musical Is Dance? The first question to pose when one wants to extrapolate musical cues from dance movement is perhaps how musical is/can be dance? That is, if there are any elements in dance movement that can be interpreted musically. The rhythmic aspect of both dance and music, and the fact that we often see dance set to music may yield a possible answer to this question. Dancing surely involves the production of rhythm. When we see dance, we clearly perceive patterns of movement that are organized in time and space. When we see dance set to music, it is common to observe that certain ways of dancing correspond to specific types of rhythmic patterns in music. Popular and folk dances illustrate that aspect very well: rumba is danced differently from samba and from swing. These dances differ from each other in the same way the rhythm of their music does. These differences pertain essentially to the metrical qualities, internal accent within the meter, and tempo of the corresponding musical rhythm. At times, the degree of synchronization attained between bodily movement and music in dance gives the impression that dance is performing spatial realizations of the music's own rhythm. These aspects suggest that there are similar rhythmic realizations in both art forms, and provide a strong motivation towards investigating the similarities between musical rhythm and rhythm in dance. In the upcoming sections of this paper, I present some of the relevant topics that reveal the similarities between musical rhythm and rhythm in dance, I discuss how can rhythm in dance movement be interpreted musically by a computer, and how this approach brings new possibilities for computer-mediated interaction between dancers and musicians in interactive dance performance. 2. ON MUSICAL RHYTHM Paul Fraisse [5] defines rhythm as "the perception of an order" (p. 151). Implied in his definition is the fact that we can predict or anticipate what will come next in a rhythmic sequence. The characteristic of predictability distinguishes rhythm from arrhythmia. He contends that the order in rhythm may be perceived or conceived. Conceived rhythm, such as the rhythm of very rapid and very slow temporal successions,1 is the rhythm in which the order is not assessed directly. All the rhythms we perceive "are rhythms which originally resulted from human [motor] activity" (p. 150). Fraisse also notes that "the possibility of rhythmic perception depends on tempo because the organization of succession into perceptible patterns is largely determined by the [Gestalt] law of proximity" (p. 151). The musical tempi we are able to perceive and produce are related to certain 1 Fraisse [5] gives as examples these rhythms the of day and night, of the seasons, or of the frequencies of light. 485

Page  00000486 periodicities of the human body such as those of heartbeats and footsteps [17]. This confers on musical rhythm a universal character, as it relates its perception and production to the physical characteristics of the human species. Richard Parncutt [18] defines musical rhythm as "an acoustic sequence evoking a sensation of pulse" (p. 453). Pulse sensation is confined to the limits of shortterm memory, which sets an upper limit to the sensation of pulse. Pulse sensations cease to exist at values exceeding approximately 1800 ms or below 200 ms [5][17][18]. The sensation of pulse, or beat, is confined to the auditory system [18]. This gives an advantage, in terms of perceptual coding, to musical rhythm over visual patterns that are rhythmically organized in time, when the periodicities in both types of organization (acoustic and visual) are within the time values in which pulse is felt. A pulse sensation arises from the complex interaction of all the periodicities involved in a given sequence. Parncutt [18] compares this situation to perceiving a musical chord in which the constituent notes seem to blend into a single sound, or the sensation of pitch while listening to complex tones. Pulse sensation is enhanced by the existence of parent or child pulse sensations that are consonant with the most prevalent one. Pulse sensation is ambiguous and a musical sequence can evoke several levels of pulse sensations. The periodicities found in musical rhythms lie between 200 and 1800 ms [3][5][17]. This range is also the one that allows for motor synchronization with a given sound stimulus. According to Fraisse [5], the optimal time intervals for motor synchronization lie between 500 and 800 ms. He also notes the striking similarity of these time intervals to certain human bodily functions and activities such as the rhythms of the heart, walking, and spontaneous and preferred tempo.3 Not surprisingly, these time values (75 - 120 beats per minute) comprise the musical tempi of most dance music. The periodicity or regularity inherent in the concept of rhythm bears, according to Fraisse [4], strong affective connotations. Fraisse notes that from perceptual standpoint there is a "satisfaction" aroused by the return of what is anticipated in a rhythmic sequence and that musical rhythm induces motion - we often respond to musical rhythm with bodily motion by rocking, tapping our feet, or dancing. Moreover, the ability of periodic temporal patterns to synchronize humans in leisurely and working activities, contributes to enhancing rhythm's affective character. Musical rhythm induces bodily movement and bodily movement can generate musical rhythm. The 2 Consonant rhythmic strata, as defined by Maury Yeston [23], are those in which the divisor for one level is a factor for the other (e.g. one level consisting of half-notes and another consisting of quarternotes). A rhythmic stratum is an isochronous level in a rhythmic hierarchy akin to Schenkerian level. 3 Spontaneous tempo, also called personal tempo, or mental tempo, is the pulse which humans tap an ischronous sequence spontaneously. Preferred tempo corresponds to the speed of a succession of sounds or lights that is judged as being neither too fast or too slow [5]. clear connection between the perception and production of musical rhythm to the physical characteristics of the human body suggests an investigation about which rhythms in dance may bear qualities akin to musical rhythms. 3. THE RHYTHM(S) OF DANCE Rhythm is a defining characteristic of dance [10][1][21][22]. Philosopher Francis Sparshott [22] notes that since everyone agrees that the "rhythm" of a dance is one of the most important characteristics, "we would expect the possibility of radical differences among dances to go with a corresponding indeterminacy in the concept of rhythm. And so it does" (p. 154). Choreographer Doris Humphrey [16] is the author who perhaps defines the existing rhythms in dance more clearly. She acknowledges three types of rhythms that are used in dance: motor rhythm, breath rhythm, and emotional rhythm. The motor rhythm, which Humphrey also calls the "beat" rhythm, is produced by the motor mechanism, and is the one she considers to be the most important for dancers: "Here is where the original dance began - with the feet - and here is where it still carries on, in the main" (p. 105). She also thinks that the awareness of accent, the energy punctuated by the beat, stems only from the change of weight in dance established by the human feet when walking. The act of walking is the key pattern of fall and recovery, it is the giving into and rebound from gravity she considers so fundamental towards providing the sense of beat. According to Humphrey, this play with gravity, primevally manifested by walking, is the very core of all movement. Breath rhythm gives the sense of phrasing and is qualitatively different from motor rhythm since it has not a sharp accent. According to Humphrey, breath rhythm is not concerned with the physical life in the sense that motor rhythm is. The idea of breath rhythm - the inhalation, suspension and the exhalation - can be transferred to other parts of the body, such as the knees, arms, or even the whole body. Finally, emotional rhythm is related to the succession of dramatic states a piece of dance may have. It may be cast into a breath rhythm, motor rhythm, or movement sequences. Its radical distinction from the previous type of rhythms is that - although there is some sense of regularity inherent in both motor and breath rhythms - the emotional rhythm is unstable "as the human being is not capable of sustaining a feeling in an absolutely steady intensity and rhythm" (p. 108). Humphrey emphasizes this aspect of emotional rhythm by asserting that "emotion, by its very nature, fluctuates; hence the dramatic rhythm pattern must show variation if it is to be convincing" (p. 108). Doris Humphrey's thoughts about rhythm in dance are particularly striking in their kinship to some of the possible "rhythms" in music. In music, one can at least discern similar rhythmic levels: a "beat" (pulse) level, that establishes a local regularity in terms of rhythmic organization; a phrasing level, also regular, establishing longer term rhythmic organizations whose relation to 486

Page  00000487 breathing has also been noticed;4 and an emotional rhythm level, perhaps not so apparent in all music styles as in dance, but bearing the fluctuating character proper to a dramatic/expressive rhythm as defined above. 4. DANCE MOVEMENT AND MUSICAL GESTURE COMPARED Paul Hodgins [11] argues that analyzing the relationship between music and dance is a difficult task since music and dance "share certain fundamental characteristics of rhythm, structure and function so innately that it is difficult even to differentiate each discipline's method of realizing such elements." (p. iii). At the temporal level, these similarities are also apparent, not only by the similarities in certain aspects of their vocabularies - phrase, breath, pause, tempo, pulse -, but also by the fact that both disciplines deal with movements of the body measured in time. However, Hodgins notes that there is a qualitative difference between the gestural tempi of music and dance. Quintessential ingredients of musical gesture such as piano trills, string tremolos, or woodwind arpeggios are executed by actions of small body parts (fingers, wrists, forearms), whereas even the smallest gestures in dance generally involve larger body parts, hence more time-consuming. Musical rhythmic action and phrasing are largely conditioned by the physical characteristics of the musical instruments. The length and shape of phrasing in instrumental music is dependent on instrumentally idiomatic elements, such as the length and speed of a string player's bow or the force and duration of a wind player's breath. In dance, the phrase's shape and length is determined by a much larger and less temporally precise instrument - the human body in motion in a dance studio or stage. The differences in timing of rhythmic action in dance and music create pervasive temporal consequences all the way up the structural hierarchy: "[t]he movement phrase [in dance] frequently translates into a larger musical structure: the duration of a period, say, or even an entire section in simple binary form" [11] (p. 14). The idiosyncrasies on the nature of rhythmic production in music and dance are in a sense linked to the size of the body parts used. Dance in general involves wider movements performed by larger body parts. Music playing involves smaller body parts and more spatially contained gestures. This aspect may explain why timing in music is in general more precise within the temporal confinements of musical rhythms. An instrumentalist is able to produce smaller pulse subdivisions when performing music, thereby giving a more precise sense of pulse and timing, since he performs more contained gestures in space and is working with smaller body parts. 4 Fisher [2] (pp. 43-60) in his extensive review about what constitutes a phrase in music provides a definition that encompasses both tonal and post-tonal music. This definition includes a physiological condition, which states that a musical phrase "corresponds to a single, complete breath" (p. 54). However, the essence of musical rhythm is heavily grounded on the physical characteristics of the human body. The relationship between musical tempi and walking, heartbeat rate, spontaneous and preferred tempo [5], establishes that there are rhythms in music that closely connect to those in dance. Walking, as considered by Humphrey [16], is at the core of movement. The accentual character in the act of walking provides the awareness of accent present in all motor rhythms, the beat rhythms. One would then easily assume that there is a region of rhythms produced in dance that intersects with those produced in music. This is what makes the act of dancing in synchronization to music possible. This may also mean that this region of rhythms in dance would be suitable for a musical interpretation. 4.1. Finding the Common Denominator Although there are similarities between the rhythmic realizations of dance and music in time, we found that certain differences were also apparent. After comparing the movement actions in dance and in music, we became aware that the phrase level in dance is in general extended over larger time spans than in music. We also became aware that dance involves more full-body actions than music and therefore the movement tempi in dance tend to span longer than those of music. When we surveyed Humphrey's ideas about the rhythms of dance, the motor rhythms - rhythms that have an accentual character that serve to establish the sense of beat in dance- seem to be the ones that can produce a series of stresses and accents that can convey the sense of pulse, in a similar fashion to what happens in music. The expressive rhythms and the phrase rhythms miss at least one of the two conditions to establish the sensation of pulse - the expressive rhythms are too irregular and the phrase rhythms, although regular, do not have an accentual character. This common rhythmic region could thus be considered to range from the pulse level in music eventually from a pulse's first subdivision for slower pulses - up to the metrical level in faster tempi in musical rhythm. At the phrase level, as pointed out by Hodgins [11], we already encounter different rhythmic behaviors in music and dance. 5. EXTRAPOLATING MUSICAL RHYTHMS FROM DANCE MOVEMENT The theoretical background presented above led to the creation of a small software library for Max/MSP called the m-objects5, which does the real-time extrapolation of rhythms from dance movement that have temporal qualities akin to musical rhythms - rhythms that have 5 In [8] I do a presentation of the whole library; in [9] I present the main algorithm for the musical rhythm extrapolation from dance movement. For more detailed information about the overall study, including a tutorial introduction to the library, see [7] or http// 487

Page  00000488 an articulatory nature and whose time-spans are between 200 and 1800 ms. Schematically, the software that was developed works as follows: a fixed video camera grabs the movement data at 25 or 30 frames per second. The video signal is digitized and the sum of pixels that changed brightness between consecutive frames is computed (a technique commonly known frame differencing). This can be done using Cyclops6 [13] or SoftVNS 2 v.motion [15] object. Subsequently, that time-varying data is given a representation in the frequency domain by the m-objects, and the musical cues are extracted from dance movement in real-time. The user of the system can map those data directly, for example, by utilizing the frequencies extracted from dance movement that have musical relevance to generate rhythmic structures, or can use the data to feed an adaptive clock and enable a dancer to control musical tempo in real time. 5.1. The Musical Processing of Dance Movement The library that was created performs a real-time spectral representation of the time-domain representation of the frame-differencing signal, and does certain types of musical analyses on this signal. The spectral representation is done by object m.bandit, which employs a bank of 150 second-order recursive band-pass filters and uses the Goertzel algorithm [6] in order to get a magnitude and phase representation of each center frequency (see Figure 1). The center frequencies of the filters span from 0.5Hz to one-half of the sampling/frame rate and their bandwidth is proportional to the center frequency (about 10% of the center frequency). The musical analyses on the signal are also performed by this object and include the calculation of the instantaneous tempo of a movement sequence and the filtering of periodicities present in the movement that exceed the 200-1800 ms range. The musical tempo detection of a dance performed by the system implements the theoretical knowledge that was gathered about pulse sensation and musical rhythm described above. If we want to determine the musical pulse in dance movement one should analyze if the periodicities in the motor articulations of dance movement evoke the sensation of regularity akin to musical pulse. That is, if these articulations bear the durational characteristics of musical rhythms, and if they are in a certain proportion to each other in a way that pulse is reinforced. The approach utilized for determining the pulse in a movement sequence is the same that can be applied for finding the fundamental frequency of a pitched complex tone. It is based on a harmonicity criterion, i.e., that the pulse sensation is enhanced by periodicities that are integer multiples of a beat duration as proposed by Parncutt [18]. The spectrum, as computed by the Goertzel algorithm, is cross-correlated with a 1 Hz pulse train7 in order to find the frequency that correlates best with the pulse train. This frequency, the fundamental frequency of the signal corresponding to the instantaneous (musical) tempo of the sequence is output by m.bandit together with the amplitudes of its first four harmonics. If we map these data directly to MIDI note events, for example, we can generate interesting musical rhythms in real time that are extrapolated from the dancer's movement. The fundamental frequency of the signal (the instantaneous tempo) is output by the object and can be used to feed an adaptive clock (object m.clock) thereby enabling a dancer to change the tempo of a musical sequence being played. The object m.clock is fed with an initial tempo to control a sequence and adapts a new tempo if the output from m.bandit falls within a predefined margin for adaptation. 6. SOFTWARE APPLICATIONS IN INTERACTIVE DANCE PERFORMANCE 6.1.A Possible Framework for Computer-Mediated Interaction in Dance The software library that was created intends to promote certain ways of computer-mediated interaction in real time between a dancer/choreographer and a composer/musician. It is designed to be operated by specialists from the fields of dance and computer music. The main goal is to allow the composer of electronic music certain types of control over the temporal articulation of the electronic music score during performance, otherwise impossible to achieve with prerecorded music. This control ranges from tempo adaptation of the score to the dancer's movement - thereby enabling the dancer to slightly accelerate or slow down the music being played - to the generation of 7 The spectrum of a 1Hz pulse train contains harmonic peaks, and the output of the spectral cross-correlation will thus favor signals that contain harmonic peaks. 0,5 Hz SR/2 Figure 1. Magnitude spectrum output by m.bandit. 6 Max objects appear in bold typeface in the text. 488

Page  00000489 musical rhythmic structures extracted from certain types of rhythms in the dancer's movement. A camera connected to the computer observes the dancer and the computer produces movement analysis data that is musically interpreted at the temporal level. The musician/composer who operates the computer receives a temporal-level interpretation of the movement analysis data in real time. It is up to the musician to decide what type of controls/parameters for interaction are given to the dancer over the musical content. The computer thus acts as a mediator in the process and is responsible for providing the musical elements extracted from movement that can be used for interaction. It acts as a sort of a filter that can extrapolate musical elements from a dance, thereby opening a channel for musical communication/interaction between dancers and electronically-generated music in real time. This type of interaction framework thus calls for a more intense participation of the composer/musician in the performance of dance with electronic music. It also opens interesting possibilities for improvisation or for the performance of open-form music/dance structures. The musician operating the computer can always change the parameters for interaction during performance and feed back different musical content to the dancer, or map the musical elements extracted from the dance to different parameters in the electronic score. 6.2. Creative Work Done So Far In the performances utilizing the m-objects, I use generative algorithms in which the pitches are articulated by the computer's (rhythmic) interpretation of the dancer's movement. During a performance, one can give or take away the rhythmic control of the music to the dancer as well as modify the output of the algorithms according to the musical result being produced. I utilized this m-objects library in four artistic collaborations so far: Etude for Unstable Time (2003 -2005), Olivia (2004), Will.O.wlsp (2005), and With Drooping Wings (2007). Etude for Unstable Time was the first piece for interactive dance created using the m-objects in 2003 in collaboration with choreographer/dancer Maxime Iannarelli and was premiered publicly in Pisa at the PLAY! concert of the Computer Music Modeling and Retrieval 2005 conference on September 26. This short piece consists of a structured improvisation in two parts. In the first part, the rhythm of music is generated solely by the computer's rhythmic interpretation of dancer's movement. In the second part there is a game played between the dancer and me over the control of the tempo of sequence of samples being played. The m-objects had their first public presentation in Olivia, a dance solo for children based on the cartoon character created by Ian Falconer. This solo was choreographed and danced by Isabel Barros, with music composed by me, and was premiered at Teatro Rivoli, in Porto, Portugal, on May 6, 2004. In this piece, I used the m-objects to create two magical moments in the show. The first was when Olivia went to the museum and pictured herself in a ballet class. When the dancer started dancing, a piano solo was generated utilizing the rhythms produced by her movement, thereby giving the impression that the character Olivia was controlling the piano music. The second magical moment happened when Olivia was having insomnia at night and started rolling in bed until eventually began to dance, generating all of the rhythm being played by her movement. Will.O.wlsp is an interactive installation conceived by Kirk Woolford in which a visual particle system is generated from captured sequences of dance movement, premiered at The Waag Society in Amsterdam on March 16, 2005. In this installation, the m-objects interpret the movement of the visual particles in order to generate the rhythms that control the sonic grains of the soundscape. This is the only piece in which all the behavior of the system is pre-programmed and no algorithms are manipulated at runtime. In With Drooping Wings, a choreography by N6 Barros for the Balleteatro of Porto (Portugal) utilizing Henry Purcell's music from Dido and Aeneas, I used the m-objects with more than one dancer moving at the same time. In this piece, I created this schizophrenic harpsichord that plays on top of Purcell's music activated by the dancers' movement on stage. Although the result is (purposefully) chaotic, there is this strange connection established by the overall movement of the dancers and the rhythms being played by the harpsichord. 7. MORE INFORMATION The m-objects and other related information are available for download at 8. REFERENCES [1] Beardsley, M. "What is going on in a dance? Dance Research Journal, 15, 1 (1982), 31-36. [2] Fisher, G. Phrase procedure in the music of Seymour Shifrin. Ph.D. Thesis, Columbia University, New York, NY, 1995. [3]Fraisse, P. Les structures rhythmiques. Publications Universitaires de Louvain, Louvain, France, 1956. [4] Fraisse, P. Psychologie du rythme. PUF, Paris, 1974. [5] Fraisse, P. "Rhythm and tempo." In D. Deutsch (Ed.), The Psychology of Music (pp. 149-180). Academic Press, Orlando, FL, 1982. [6] Goertzel, G. "An algorithm for the evaluation of finite trigonometric series. American Mathematics Monthly, 65 (1958), 34-35. [7] Guedes, C. Mapping Movement to Musical Rhythm. A Study in Interactive Dance. Ph.D. Thesis, New York University, New York, NY, 2005. [8] Guedes, C. "The m-objects: A small library for musical rhythm generation and musical tempo 489

Page  00000490 control from dance movement in real time." In Proceedings of the International Computer Music Conference, Barcelona, Spain, 2005. [9] Guedes, C. "Extracting musically-relevant rhythmic information from dance movement by applying pitch-tracking techniques to a video signal." Proceedings of the Sound and Music Computing Conference SMC06, Marseille, France, 2006. [10] Hanna, J. To Dance is human. A theory of nonverbal communication. University of Texas Press, Austin, TX, 1979. [11]Hodgins, P. Relationships between score and choreography in twentieth-century dance. Music, movement and metaphor. Mellen, London, 1992. [12] html [13] tml [14] [ 15] [16] Humphrey, D. The art of making dances. Grove Press Inc, New York, NY. 1959. [17]Parncutt, R. "The perception of pulse in musical rhythm." In A. Gabrielsson (Ed.), Action and Perception in Rhtyhm and Music (pp. 127-138). Royal Swedish Academy of Music, Stockholm, 1987. [18]Parncutt, R. "A perceptual model of pulse salience and metrical accent in musical rhythm." Music Perception, 11(1994), 409-464. [19]Rowe, R. Interactive music systems." Machine listening and composing. MIT Press, Cambridge, MA, 1993. [20]Rowe, R. (2001). Machine musicianship. MIT Press, Cambridge, MA, 2001. [21]Royce, A. Movement and meaning. Creativity and interpretation in ballet and mime. Indiana University Press, Bloomington, 1984. [22]Sparshott, F. A measured pace." Toward a philosophical understanding of the arts oJ dance. University of Toronto Press, Toronto, Canada, 1995. [23 ]Yeston, M. The stratification of musical rhythm. Yale University Press, New Haven, CT, 1976. 490