THE "CONDUCTOR'S JACKET":
A DEVICE FOR RECORDING EXPRESSIVE MUSICAL GESTURES
Teresa Marrin and Rosalind Picard
Affective Computing Research Group
Media Laboratory
Massachusetts Institute of Technology
Cambridge, MA 02139 USA
marrin, picard@media.mit.edu
Abstract
We present the design and architecture for a device called the "Conductor's Jacket," which we built to collect and
analyze data from conductors. This device is built into normal clothing and designed in such a way as to allow for
normal activity and not encumber the movements of the wearer. The jacket records physiological and motion
information from musicians in order to better understand how they express affective and interpretive information
while performing. We present our design considerations and some preliminary results from data collection trials that
we conducted on a range of professional and student conductors in 'real-world' situations.
1 Introduction
A common issue encountered in the development of new instruments is the question of how to map the input control
stream (often in the form of transduced gestures) to the output audio stream. That is, how to generate complex
musical responses that coherently reflect the intentional information contained in gestural inputs. Often, designers of
interactive performance systems choose mappings that apply one continuous axis of the input control stream to a
single continuous parameter at the output stream. This method often yields a clear and useful result, but has
limitations. For example, if there are characteristics or patterns in the performance of the gesture that require
interpretation, then more complex methods are required.
We are interested in building a system that recognizes and responds appropriately to expressive features in musical
gestures. The first phase of this project involves designing and building a reliable system to collect data from
musicians in real performance situations. The second phase involves analyzing the data to learn features that are
natural indicators of musical expression. Both of these phases are aimed at learning more about how musicians
naturally modulate their gestures and physiology when they perform music, so that better mappings can be developed
between the input gestures and the output audio stream. The final phase of this project will involve constructing such
mappings for the system described below.
We chose to focus on conductors because they use a highly specific code of gestures in order to indicate the expressive
and interpretive elements in the musical structure. Additionally, conducting gestures often indicate more global
phenomena than the individual note events and are not constrained by the requirements of performing on a
mechanical instrument.
The data-acquisition system described in this paper, which we call the "Conductor's Jacket", is a wearable array of
sensors, communications, and computation, which detects and records physiological changes and physical motion
without interfering with the way that the gesture is naturally performed. This approach builds upon experimental and
analytical techniques developed for situations involving human affective communication [Healey and Picard 1997,
Picard 1997]. We present the system architecture and design considerations below, along with preliminary results
from use of this system by three professional conductors and four conducting students.
0
