Page  00000001 The Performer-Instrument Interaction: A Sensory Motor Perspective M. Sile O'Modhrain and Chris Chafe Center for Computer Research in Music and Acoustics (CCRMA) Music Department, Stanford University, Stanford, CA 94305 sile, cc Abstract In this paper, we present the results of a study designed to discover how players internalize the dynamic behavior of a new musical instrument. Ten experienced musicians played a series of short melodies on a virtual Theremin capable of haptic as well as auditory feedback. Participants were assigned to one of two force conditions, viscous damper or spring. Results indicate that the viscous damper took players longer to learn. In the end, however, players in both conditions performed equally well, indicating that they had learned to compensate for differences in the dynamics of the instrument. Moreover, when all force feedback was removed, players exhibited inverse behavior showing that they had indeed internalized the dynamics of the instrument. 1 Introduction In a recent study (O'Modhrain and Chafe, 2000), the authors examined the effect of adding force-feedback to a computer-based musical instrument. Experienced musicians played a series of short melodies on a virtual Theremin whose controller was capable of relaying forces to the musician's hand. The position of the player's hand in space was sampled continuously, and this data was analyzed to judge the effect of different force conditions on playing accuracy. Our hypothesis that adding force feedback should improve accuracy was born out statistically. Results showed that force feedback which was better correlated with changes in the parameter being controlled (pitch) made the task easier. One question which arose from this previous experiment is whether the differences in the various mappings imposed on our virtual Theremin are providing feedback which is of long-term use in performance or whether players treat them as artifacts of the interface which they will overcome given sufficient practice. As Shadmehr has shown (Shadmehr, 1994), humans excel in their ability to adapt rapidly to the variable dynamics of their arm as their hand interacts with the environment. Given sufficient practice in a novel force field (approximately 700 discrete reaching movements), he demonstrated that his subjects had internalized the dynamics of a force field such that, when all force feedback was suddenly removed, trajectories of hand movements showed clearly that subjects were expecting to encounter certain forces at particular points and had "planned ahead" to account for these disturbances. Based on these findings, we conjectured that given sufficient practice, players would "learn" the mapping between the haptic and auditory response of the virtual Theremin and learn to compensate for any feedback which was impeding their performance. We hypothesized that, if players were given long enough to play in a given force condition, they would adapt to it and the differences between the spring and viscous damper feedback conditions would be greatly reduced or disappear altogether.

Page  00000002 2 Experiment 2.3 Procedure Ten volunteers, none of whom had taken part in the previous experiment, were recruited from the faculty and student population at Stanford's Center for Computer Research in Music and Acoustics (CCRMA.) As with the first experiment, all were experienced musicians and none had any known hearing or motor impairments. 2.1 Apparatus Our experimental apparatus consisted of a haptic display device, the "Moose," and a PC with a software MIDI synthesizer (Gillespie and O'Modhrain, 1997). The various force conditions were generated in software in real time. Sample melodies were "played back" via MIDI, while the pseudo-Theremin's sound was produced by the PC's internal speaker (since this produced the most realistic Theremin sound and was also spatially separated from the MIDI synthesizer sound output, making it easier for participants to separate the two audio sources). A computer-controlled metronome, synchronized with the tempo of the current melody, ensured rhythmic accuracy. 2.2 Stimuli The melodies used were opening phrases of melodies taken from the Themefinder database, maintained by the Center for Computer-Assisted Research in the Humanities (CCARH), at Stanford ( The melodies chosen were diatonic, ranging in length from 9 to 16 notes with an average of 12 notes. Melodies contained no rests or directly repeated pitches. All contained both a rising and falling perfect fifth interval. Participants were randomly assigned to one of two experimental groups - positive spring feedback or viscous damper feedback. All participants played all 18 melodies. Melodies were presented in random order with one exception: melody 18 was used as the training melody and was presented again at the very end of the experiment. Participants were given an initial period of time to get used to the device. An arpeggio and a control melody (melody 18) were then recorded and represented baselevel performance in the force feedback condition to which the subject was assigned. Participants then completed 17 further trials from the melody set, followed again by melody 18 with force-feedback, and then melody 18 with no force-feedback. For each trial, a "first attempt" at the melody was recorded; Subjects then practiced 4 times before a second recording was made. 3 Results 3.1 Convergence of Force Conditions An independent-samples t-test with force condition as the independent variable and improvement in performance of the control melody as the dependent variable revealed no significant difference between the two force conditions tested. Thus, taking the control melody as a rough measure, we can conclude that players did become accustomed to the mapping between haptic and auditory response for the instrument and the differences between viscous damper and spring conditions seen in the experiment reported in (O'Modhrain and Chafe, 2000) disappeared. 3.2 Adaptation Curve In order to estimate "playing accuracy," we scored each performance against a computer-generated template for the appropriate melody. Each score represented the average RMS error across the whole melody for that performance. Each participant was recorded twice in their attempts to play each of the 18 melodies and the best score for each melody was kept as the score for that trial. Figure 1 is a plot of these scores for one participant who was assigned the viscous damper condition. To gauge the slope of adaptation over the duration of the experiment for each force condition, we divided each participant's 18 trials into 6 groups of 3 trials. We averaged each group of three trials for each subject

Page  00000003 30 21) 1D I I I I I I I I I Figure 1: RMS error of one subject's 18 melody trials with viscous damper force condition (best score of 2 takes for each melody). individually, to obtain a graph for performance over time. Finally, to see if there was a difference in the rate of adaptation between the two force conditions, we plotted the mean scores for each group of trials by force condition, Figure 2. With this plot, there appears to be no significant difference between the curves for the two force conditions. Both drop steeply at their start and level out after the second bin. Figure 2: Grouped trials by force condition, averaged across participants (3 trials per bin). ~tda per E3spri ng jo 71) 51) 51)!ID 4 Discussion In the previous study, we had observed a significant difference between performances in the spring and viscous damper force feedback conditions. The spring, where force feedback is coupled to position, faired better than the viscous damper. Why, then, is the difference between spring and damper conditions seen in the first experiment not visible here? The answer lies in the curves for each force condition over the earliest trials. Figure 3 plots mean score by force condition for the first 7 melodies played. It can be seen from the slopes in Figure 3 that play Figure 3: First 7 trials by force condition, averaged across participants.

Page  00000004 ers did indeed adapt to different force conditions at different rates, confirming the hypothesis for this experiment. What was surprising was that this adaptation occurred within the first few minutes of playing and that the viscous damper feedback condition took players only a few trials longer to adapt to than the spring condition. The differences in performance for these force conditions observed previously was therefore a function of the fact that the period of time for each trial was too short for players to adapt to the viscous damper condition but long enough to allow them to adapt to the spring. Finally, to be certain we had forced players to internalize the dynamics of the force condition in which they played, we removed all force feedback at the very end of the experiment and recorded an encore performance of the training melody. Figure 4 is a plot showing the template for the training melody (the solid line), the final performance in the assigned force condition (the solid thin line) and the encore performance once all force feedback had been removed (the dotted line.) The force condition assigned to this player was the viscous damper. It can be clearly seen that, when force feedback was removed, all large leaps were over-estimated while small intervals remained more or less the same. 5 Summary and Conclusions The results of this study lead us to conclude that, like Shadmehr, we had caused players to build an internal model of the dynamics of the instrument, the inverse of which became apparent once force feedback was removed. The present studies have provided empirical evidence supporting the hypothesis that adding haptic feedback to interfaces for computer-based musical instruments improves the player's ability to control these instruments. Moreover, it has been shown that those force conditions where haptic feedback was closely correlated with auditory feedback resulted in better initial performance than conditions where auditory and haptic feedback were not correlated. Given practice, however, players can learn to compensate for inappropriate mappings between the haptic and auditory responses of the instrument, though the learning 500 400 -300 -200 0 1000 2000 3000 4000 5000 6000 7000 8000 time Figure 4: Comparison of the same melody before and after training. curve for inappropriate mappings is longer. These results, obtained within the context of a musical performance on a virtual Theremin, indicate that mappings which support the objective of the task (in this case accurately locating points in space) can accelerate the initial phase of learning the behavior of an instrument. 6 REFERENCES Gillespie, R.B. and O'Modhrain, M.S. 1997. "The Moose: A Haptic User Interface for Blind Persons." Proc. of the 3rd Annual WWW6 Conf Santa Clara, California, Also available as STAN-M-95, CCRMA Tech. Rep. Music Dept., Stanford University. O'Modhrain, M.S. and Chafe, C. 2000. "Incorporating Haptic Feedback into Interfaces for Music Applications." Proc. of the Eighth Intl. Symp. on Robotics with Applications. Maui, Hawaii. Shadmehr, R. and Mussa-Ivaldi, F. 1994. "Adaptive Representation of Dynamics During Learning of a Motor Task." Journal of Neuroscience 14 (5, Pt 2): 3208-3224.