Page  00000001 Music and Gesture: Sensor Technologies in Interactive Music and the Theremin based Space Control Systems Andrei Smirnov Theremin Center for Electroacoustic Music at Moscow State Conservatory Bolshaya Nikitskaya 13, Moscow 103871, Russia theremin@dialup.ptt.ru "A musical instrument is a device to translate body movements into sound." --Sawada, Ohkura, and Hashimoto [1, 3] This work attempts to give the brief overview of some of the most important, but partially forgotten biomechanical concepts based on the techniques to measure body movement of workers, athletes and musicians, developed in Russia in 20-s. These experimental works and their results are very useful now to study musical performance from the point of view of the development of Interactive Music Systems, based on different approaches to measure the human body movement and to create different Gestural Interfaces to control the musical performance. Mainly the work will be focused on the concept of Alive Motion developed by Russian psychologist Nikolai Bernstein. Expressive musical gesture and language One of the possible approaches to describe the expressive musical gesture is based on the hypothesis on how to separate parameters of motion down to their smallest discrete segments, from which it is possible to extrapolate higher-level features and map complex musical intentions. This approach based on the analogy between gestural and verbal languages and also makes a parallel with the gestural language of conducting and its analysis and applications to software mappings [1]. T.A. Marrin gave the definition of gesture as an openended set of physical possibilities for communication; It is a reconfigurable set of parameters similar to those of the spoken sound. By successively defining all the parameters at each subdivision (down to their most basic atoms - the "phonemes" of gesture), one begins constraining the different layers of meaning (languages) that can exist within it. Languages are, by definition, hierarchical structures of meaning in a particular communication medium. That is, they are complicated sets of mappings between concrete objects and their representations [1]. At the same time, T.A. Marrine emphasized, that the musical expressivity is one of the most complex of human behaviors, and also one of the least well understood. It requires precise physical actions, despite the fact that those very actions themselves are hard to model or even describe in quantifiable terms. Music and gesture are both forms of expression which do not require spoken or written language, and therefore are difficult to describe within the constraints of language. [1] In my conversations with Russian acoustician Alex Golembo regarding touch and piano sound expressivity, he mentioned one interesting observation: piano players and listeners often have different understanding of the piano sound and exact moment of it s beginning. From the listener s point of view, sound begins since the moment, when the hammer hits the string, from the players point of view, since the moment, he started to move towards the piano key. The term "touch " means the way the pianist touches the key or, in other words, what was happened just before the pianist pushed the piano key. But from the point of view of piano mechanics, all special movements of the pianist on his way to the piano key can t have any effect on the sound produced. Thus what is piano touch and does it exist at all? Fig. 1 Cyclogram of the piano touch measurements at GIMN, Moscow 1925. Loops above the hands of wired up pianist trajectories of lamps on his hands, printed on the photo plate. It is a stereo photograph to get the spatial information about trajectories. Nikolai Bernstein and his concept of Alive Motion Since 1922, when the Central Institute of Labor was founded in Russia by poet and scientist Aleksei Gastev. Numerous techniques have been developed there to measure body movement of workers, athletes and musicians. The most important biomechanical research of

Page  00000002 the human body movement was initiated there by Russian psychologist Nikolai Bernstein. "It is necessary to point out, wrote in 1930 Nikolai Bernstein, that not only methods, but also concept of rationalization of motions is far from being as simple, as was thought earlier. The simple struggle of Taylor, and later Gilbert with superfluous motions and comprehension of biomechanical operation as simple sum of series of motions, which one is possible to riddle as a grain on sorting, starts to succumb the place to comprehension of a motion complex as organically unseparable whole, always recalling on change of any one part, by realignment of all remaining ". N.A.Bernstein, On the construction of motions [8, 9, 10] Methods of registrations and analysis of motion, developed by Bernstein, and studies, conducted on their bases, permitted him to formulate a number of the most important statements. Main of them is, that the motions of a living organism should be surveyed as some kind of organs with the properties, intrinsic to anatomic organs: At first, the alive motion reacts, secondly, it possesses the quality of the regular evolution and involution. [8, 9, 10] At analysis of a reactivity of motion he has found out its selectivity. It has resulted him in the concluding, that "the motion is not a chain of parts, but the structure, differentiated on parts,- a holistic structure, possessing at the same time the high level of differentiation of the elements and varied-electoral form of relationships between them". [8, 9, 10] Thus alive motion, according to Bernstein, it is a reactive, developing functional organ, possessing the differential structure and an own biodynamical tissue. Under a functional organ it is necessary to understand any temporary combination of forces which are capable to execute definite achievement, and it is absolutely not necessary to bind the idea of an organ with representation about morphological, statically constant formation. Such a complex formation, as an alive motion, should have definite vital signs, to define which Bernstein used the concept of "a propulsion problem". The problem of "the construction of motion" in a unique subject situation is extremely complex. To solve it, body, possessing mentality, is compelled by any nonrational way to comprehend the complex physics (statics, dynamics, kinematics) of a particular subject situation and to compound it with a solid bodily machinery The alive motion is less of all similar to mechanical movement of a body in space. Bernstein on the basis of generalizing all set of topological and metrical properties of a motility in its relationships with external space has entered the concept of a motor field, in which one the topology dominates above the metrics. The careful analysis of a drawing of motions even well learnt and multiply replicating in the similar situations, testifies to their uniqueness. The biodynamic tissue of the motor act as is unique, as a fingerprint. According to Bernstein, any exercise is a repetition without repetition. Any operating never repeats, but always is under construction. At construction of operating it is always possible to observe the competition of its conservative properties, defined by already existing programs and mnemonic schemes, and its dynamic properties, defined by the novelty of a situation, novelty of the purposes and senses of the arisen propulsion problem [10]. For example, using biomechanical techniques to study musical performance M. Lammers and his colleagues looked for factors which differentiate the highly successful performer to answer the frequently asked question of trombone players by non-musicians is, 'How do you know where to put the slide?'. In summary, examination of the pedagogical material on trombone performance makes it clear that there is not agreement on where the slide should be placed. This may not be surprising since experienced performers can alter pitch without changing slide position. This allows them to play correctly even when the slide is out of position. Nonetheless, it still remains to be seen how far performers really do move the slide [7]. Chronotop The comprehension of alive motion can be essentially facilitated, if we shall describe its own Chronotop, i.e. space (topos) and time (chronos), in which one it exists. It is possible to characterize the chronotop of alive motion as an active. In chronotop space and time exist as unity of contrasts. Space is the halted time, and the time is a driving space. The unity of space and time in chronotop means a capability of their transition into each other. It does not exist in each separate instant, therefore it is difficultly representable. In each separate instant the spatial characteristics of chronotop undefinable. For their definition it should be bodily organized in time. But this means, that even before such deployment in some gaugings it exists bodily and comprises the program of the substantial deployment. Thus chronotop, existing in the latent form, should contain everything indispensable and sufficient for its deployment in space and time (in a subject world). To realize that, chronotop of alive motion should contain its own, internal, subjective space and time, which are at the same time as substantial, as objective space and time. Only on their basis the problem of mastering the space and time of life could be resolved. That is the main paradox and puzzle of the chronotop of alive motion [10]. Theremin Sensors In 1919 Russian inventor Leon Theremin invented the theremin, an electronic musical instrument played by free movement of the performer's hands in the space surrounding it. It was the first electronic musical

Page  00000003 instrument to operate via wireless, non-contact sensing. It has also been called the "first truly responsive electronic musical instrument," because it allowed an enormous amount of direct control to the user. Its musical mapping was simple and straightforward - a linear, one-to-one correspondence between proximity and pitch (in the right hand) or loudness (in the left hand) - but its results were astonishing. "Few things since have matched the nuance in Clara Rockmore's lyrical dynamics on this essentially monotimbral, monophonic device." [11] Kavina playing the therpsitone, the I theremin based t dancing platform at the Theremin Center, Moscow 1996 In the early 30-s Leon Theremin invented terpsiton, the special dancing platform with the flat theremin antenna, built in it, which we can consider as the first music / gesture interface. The Theremin and Terpsitone provided a new paradigm for musical instruments, and their contribution as an interface were profound. With the current software, we are limited to detecting rudimentary positions and a few general behaviors like jaggedness or velocity. Ideally, one could imagine detecting more complex composite gestures (more akin to sign language) in order to create a 'lexicon' that would allow users to communicate with systems on a much more specific level than they can presently. Imagine, for example, being able to point to individual instruments in a 'virtual' orchestra, selecting them with one hand and controlling their articulation and volume with the other.' [1, 4] -- David Waxman, "Digital Theremins," Classical theremin development and training is a part of the Theremin Center activities now as well as the development of the different theremin based Space Control Systems to determine distance, speed and acceleration of the moving human body. Different theremin sensors were developed, according to modular principle to construct any desirable installations and stage sets for applications in Music and related arts. Several sensors could be used to feel the big space or to provide two or three dimensional information. The main features of the sensors: - extended sensitivity (up to 2-3 meters) - high stability - portability and low weight - compatibility with the standard computer hardware Wireless system, based on so called the Space Probes are under development now. Fig. 3 Jana Aksenova playing with the theremin sensors at the Theremin Center, 1999. In real performance the distance between antennas could be up to 3-4 meters. Using the MAX/MSP software numerous algorithms (abstractions) were created to determine and analyze the distance, speed, acceleration of the moving human body to provide the tempo following (by tracking the periodical processes) and to interpret the results into the sound synthesis or processing of external signals. The following parameters, based on velocity, recommended by T.A. Marrin [1] were found very useful: - instantaneous velocity (or current speed), the most straightforward of the three, which is obtained by approximating the first derivative of the position coordinate by using the position measurement of the previous millisecond. - The second kind of velocity is the average velocity over a small time-scale, which is taken by averaging the instantaneous velocity over a small, sliding local window. (It should be noted that using this sliding local window is equivalent to a low-pass filter, smoothing the curve and taking out the noise and jitter which might be caused by muscle tension or noise in the signal from the sensors.) - Thirdly, there is the average velocity over an entire piece or section (which is taken by approximating the first derivative with respect to over a much larger window) gives a general sense of the activity and energy level of the entire piece. [1] Several studies (so called Sonochronotops ) were created by the author to explore the possibilities of musical performance in the frame of the Alive Motion concept in artistic field. The core idea of the Sonochronotops #1 5 was to extend the existing acoustical environment by means of tracking the human body movement in the space and interpretation of the results into the construction of the new, active acoustical environment, incorporating some

Page  00000004 extraordinary acoustical features, impossible in the real acoustics, but easy and responsive enough to be mustered by the performer. Sonochronotops #1-3 were based on the extension of the acoustical features of the wind instruments (trumpet, clarinet and saxophone), played by Konstatin Adjer in front of two theremin sensors. Player could control the sound processing by means of his body, including the tracking of the temporal characteristics of his movements. Sonochronotops #4-5 (inspired by the Alvin Lucier s music) were created for performers or dancers. In these studies it is possible to change the resonant properties of the existing space by means of body position and movement, to play with the numerous acoustical feedbacks and to conduct the developing sonic structure. Acknowledgments I would like to thank Wolfgang Heiniger for the inspiring discussions that provided many stimulating ideas, Natalia Smimova for psychological information, books and important conversations. This work was supported in part by a grant from the Open Society Institute (Soros Foundation), Moscow. References 1. Teresa Anne Marrin, Toward an Understanding of Musical Gesture: Mapping Expressive Intention with the Digital Baton, Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial fulfillment of the requirements for the degree of Master of Science in Media Arts and Sciences at the Massachusetts Institute of Technology 2. Michael Hawley, "Structure Out of Sound," Ph.D. Thesis, M.I.T. Media Lab, September 1993. 3. Sawada, Ohkura, and Hashimoto, 1995. Sawada, Hideyuki, Shin ya Ohkura, and Shuji Hashimoto. Gesture Analysis Using 3D Acceleration Sensor for Music Control. Proceedings of the International Computer Music Conference, 1995. San Francisco: Computer Music Association, 1995. 4. Waxman, David Michael. "Digital Theremins: Interactive Musical Experiences for Amateurs Using Electric Field Sensing." Masters' Thesis, M.I.T. Media Laboratory, September 1995. 5. Cadoz, Claude. "Instrumental Gesture and Musical Composition." Proceedings of the International Computer Music Conference, 1988. San Francisco: Computer Music Association, 1988. 6. Darrell, Trevor and Alex. P. Pentland, "Recognition of Space-Time Gestures using a Distributed Representation," M.I.T. Media Laboratory Vision and Modeling Group Technical Report number 197. Appeared as "Space Time Gestures," Proceedings of the IEEE Conference, CVPR, 1993. 7. M. Kruger, M. Lammers & R. Fuller, Biomechanics of music performance: moving the trombone slide, Article for the International Biomechanics of Sport held at Sheffield, England in 1996 8. N. A. Bernstein, On the construction of the motions, Moscow, 1947. In Russian. 9. N. A. Bernstein, Studies on physiology of motions and physiology of activity, Mosco w, 1966. In Russian. 10. N.D. Gordeeva, Experimental psychology of the executive movement, Moscow: Trivola, 1995. In Russian. 11. Paradiso, Joseph and Neil Gershenfeld, "Musical Applications of Electric Field Sensing", Scientific American (pre-print), vol. 274 no. 4, April 1996 12. Mulder, Axel, Getting a Grip on Alternate Controllers: Addressing the Variability of Gestural Expression in Musical Instrument Design. Leonardo Music Journal, Vol. 6, pp. 33-40, 1996