Page  76 ï~~Tactile Audio Feedback Chris Chafe Center for Computer Research in Music and Acoustics, Department of Music, Stanford University, Stanford, CA 94305 Abstract Vibrotactile feedback is common in handheld musical instruments. Useful cues received by the player via this sense are presented, and a design which incorporates vibrotactile feedback in new controllers for physical models is proposed. I. Introduction Quite some time ago, in many forms, humans evolved a medium of expression through whacking, plucking, blowing and bowing various acoustic and mechanical oscillators. Adept control of the vibration of these systems is as necessary to music as it is to precise vocal communication. Like the voice, handheld instruments are intimately connected with our sense of touch. Primary feedback arrives by ear, but the feel of crafting a sound from a brass, wind or stringed instrument is an important secondary sense and is learned early on in training. Resistances and "give" are felt kinesthetically and vibration arrives directly through the tactile sense. Contact points in the cello, for example, are five: two legs, two hands and chest. These points register motion that adds to the player's sense of the inssrument's response to controlling gestures. II. Background The psychophysics of the vibrotactile sense has been described at length in the literature. Verrillo in [1] presented a review of the field framed by a discussion of issues relevant to musical performance. These general concepts are of importance in the following discussion: " The fingers are among our most sensitive sites. " Frequency response ranges from near 0 to approximately 1000 Hz. " Frequency discrimination is very poor. " The subjective sensation changes across frequency bands. - 3 secs. 1 4 1,, "v"v v v, -" vv_ Figure 1: Tone 1 amplitude plots are from an 880 Hz. natural harmonic played on the celletto. Top: bridge transducer. Bottom: left-hand index finger accelerometer. Amplitude sensitivity measured with sinusoidal stimuli varies with location. It is suggested that high sensitivity, such as has been measured at the fingers, is in relation to representation area in the somatosensory cortex [2]. Four independent physiological channels are known and are separable with regard to amplitude and frequency sensitivity [3]. Differences may play a role in bracketing vibrations of an instrument into distinct cues, as will be shown to occur below. A mound-shaped curve describes overall cutaneous frequency sensitivity between 0.3 and 1000 Hz. The region of best sensitivity extends from about 100 to 500 Hz. Frequency discrimination is poor compared with the ear: The finger is only able to detect differences on the order of 20 or 30%. The quality of the sensation changes from a localizable "buzz" below about 100 Hz. to a diffuse, smoother sensation for higher frequencies. From various earlier studies, it can be concluded that only certain musical dimensions are representable, specifically 8A.4 76 ICMC Proceedings 1993

Page  77 ï~~2 kHz 0.9 secs Figure 2: Tone 1. Low frequency components are shown in a spectogram of finger vibration at note onset. These components disappear as the stable oscillation sets up (seen after the cursor mark). Figure 3: Tone 2. Amplitude plots are shown of first episode (see text) of a note onset at 110 Hz. Top: bridge transducer. Bottom: left-hand index finger accelerometer. timing, amplitude, and spectral weighting, but not precise pitch. III. A Cellist's Left Hand Recordings were made of cello tones to discover some functional vibrotactile cues. The fingertip was chosen for an initial site because of its good sensitivity. Tones were played arco on the celletto, an electronic cello, and two channels were digitized simultaneously at a sampling rate of 44.1 kHz. Output from the instrument was recorded with a bimorph piezoceramic bridge transducer designed by Max Mathews. Finger motion was obtained from an accelerometer (PCB model 330a) affixed to the nail of the index finger which was stopping the string. The following analyses demonstrate two cues by which a cellist senses stability of oscillation. Tone 1 is a high pitched natural harmonic (played with the finger stopping the string lightly, not fully to the fingerboard) and sounding 880 Hz. (A5), played on the instrument's first string (A3). Tone 2 is fully stopped at 110 Hz. (A2) and played on the third string. Both tones were played with intentionally long bow attacks to exaggerate the note onset transient. 2 kHz. ":_......."................i ".. A \ ' ':".;..........: Y.;.:,. i;.:,,. Â~,.............. ". "+...:...........................-..-....... ".................."...................... "......................... <..... h": t. ~.....:..:", 0.9 secs Figure 4: Tone 2. The spectogram shows the'progression of finger vibration from note onset to stable Helmholtz motion. The pitch starts high (sporadic episode), dips below (unstable Helmholtz episode), and then arrives at the final fundamental pitch. Tone 1 is pitched above the upper frequency limit for sensation. Continuous vibration is only felt for tones pitched below about a perfect fifth lower than this note. Despite this fact, transients are still felt at pitches that are too high. Note onsets, bow direction changes and abrupt stops are sensed as brief vibrations at the fingertip. Dual amplitude plots of Tone 1 in Figure 1 contrast the output signal and the signal that passes through the fingertip. From the noisy onset into strong stable oscillation the bridge waveform shows amplitude growth. However, the fingertip recording shows a low-pass filter response: diminishing amplitude as the oscillation locks in on a pitch that is too high. During the transient, low components that are in the region of sensitivity are transmitted through the finger. These are seen in the spectogram of Figure 2. At this pitch, the cellist has a cue that discriminates transient events from stable oscillation through presence vs. absence of vibration. Tone 2 is pitched to lie with at least 7 of its harmonics in the region of sensitivity. Continuous vibration is felt through the entire course of this tone. The transient portion is still sensed as a discrete event though the cueing signal is different. As pointed out above, vibration quality can change at around 100 Hz. as it does here,,from a "rough and aperiodic" transient to a "smooth and regular" stable oscillation. Furthermore, the note onset itself consists of two distinct episodes before leading to stable, periodic Hehmhohtz (stick/slip) motion. Initial sporadic releases are followed by an interval of very unstable Helmholtz motion with a flat pitch. The sporadic release episode shows a surprising difference between the two recorded channels: Where the bridge sees only isolated releases, the fingertip feels a plucked periodic vibration at 123 Hz.(B2) one whole tone higher than the actual pitch. The phenomonen results from the bow hair sticking to the string immediately after a quasi ICMC Proceedings 1993 77 8A.4

Page  78 ï~~pizzicato, causing the string to be split into two portions. The felt pitch corresponds to the string length between fingertip and bow. Confirmation was made by studying the resulting pitch at different bow contact positions. Figure 3 shows the first episode. During the second episode, the tone exhibits flattened, aperiodic Helmholtz motion. The fingertip waveform is more complex because of competing, incommensurate oscillations. In Figure 4, Helmholtz motion (pitch A2) takes over from the earlier plucked string motion (pitch B2) and brings in lower components. Both tones confirm that fingertip vibration (or lack of vibration) can be used to gauge the time and length of articulation. Depending on the note played (pitch, note fingering, etc.) finger motion was found to provide cues through amplitude and spectral content. The player interprets cues in relation to the specific note. For example, the same message concerning oscillation stability will be received as presence/absence or smoothness/roughness cues, depending on pitch height. IV. Physical Models With the advent of physical models for synthesis, the world of electronic sound generation has a new class of "unpredictable" instruments. The same unpredictability is found to some degree in most traditional musical instruments and is easily summarized as the "french horn problem." Unruly overblown notes on the horn are an extreme example of an oscillation going one way when the performer wishes to go another. The family of real time physical models developed at CCRMA exhibits this independence in all its members - it is an article of faith in the theory of oscillating nonlinear systems that this is "a feature, not a bug." Incorporating vibrotactile feedback addresses specifically the problem of performing on instruments that are not purely deterministic. Controllers that have been attached to CCRMA's physical models include MIDI modulation wheels, MIDI keyboard aftertouch, mouse-controlled computer panels and homemade gear such as Cook's WhirlWind instrument [5]. The hand controlling a synthesis parameter locates a particular value either be ear or combined with a coarse sense of position (which may depend also on thle eye watching a cursor). Position itself is relatively coarse compared to the model's sensitivity to some parameters. Wrorse yet, the models often do not respond identically to a precisely repeated parameter value, since system state interacts with reponse inl the physical modeling world. The models exhibit multiple possible regimes of oscillation for a given set of parameter values. The electronic french horn problem is p)resently much worse than the natural one. The lip tension Figure 5: Tactile Audio Feedback is used to control the lip tension parameter of the brass physical model HosePlayer. parameter of CCRMA's nearest model, HosePlayer, determines in part which overblown note will sound. Using the controllers listed above (with a hand controlling lip tension) we have yet to hear anyone play Taps (a bugle call) without a mistake. The only possible feedback is crossmodal (ear/finger). Normally, when playing a brass instrument, lip tension control and the lip reed producing the sound would be intimately associated. Effort injected into the oscillator would be metered directly by vibrotactile sensation at the point of excitation. Instead, an electronic controller is employed which is either "dead" in this sense or imparts vibration and resistance of its own kind, and which are not derived from the oscillating system. V. Tactile Audio Feedback An inital test has been performed to see if the situation improves with addition of vibrotactile feedback by creating a direct control loop at the finger tip. The setup is diagrammed in Figure 5. Depressing a flexible metal bar corresponds to a change in lip tension. Audio output of the model is fed back to a voice coil actuator that vibrates the metal bar. With the finger depressing the bar and feeling the output of the oscillation, adept maneuvers of lip tension are possible (Taps is much more playable). Turning off audio feedback to the actuator removes vibrotactile feedback and causes the situation to revert back to imprecision. Most of the pitches are above frequency cutoff for the vibrotactile sense. Figure 6 shows a spectogram of a portion of a 8A.4 78 ICMC Proceedings 1993

Page  79 ï~~4 kHz [41 P. Cook. "TBone: An Interactive WaveGuide........... Brass Instrument Synthesis Workbench for the NeXT Machine," Proceedings ICMC 91, Montreal, pp. 297-299: 1991 15] P. Cook. "A meta-wind-instrument physical model,..............................and a meta-controller for real time performance control," Proceedings ICMC 92, San Jose, pp. 273. 276: 1992 0.45..... [6] B. Gillespie. "The Touchback Keyboard," ProceedFigure 6: A spectogram of a lip tension glissando ings ICMC 92, San Jose, pp. 447-448: 1992 shows subharmonic components when the pitch [7] C. Cadoz, L. Liswoski, and J-L. Florens. "Modshifts to the next higher harmonic. ular Feedback Keyboard," Proceedings ICMC 90, Glasgow, pp. 379-382: 1990 harmonic glissando produced by changing lip tension, as synthesized with Cook's TBone program [4]. Brief bursts of energy support subharmonics that lie below cutoff at note transitions and harmonics of subharmonics are visible. The finger on the controller experiences these moments as "buzzes" or "bumps" when the overblown harmonic changes but feels nothing of the sustained tones between..Feedback to the performer consists of the same cue as Tone 1 in the cello analysis above. VII. Conclusion Two vibrotactile cues have been explored. Certainly the number of cues is larger when taking into account the full range of an instrument's sonic possiblities. Feedback concerning oscillation timing and quality has been found. The experiment incorporating vibrotactile feedback in the controller for a realtime physical model of a brass instrument can be extended simply in more sophisticated controllers: Feed the audio output of the synthesis back to the controlling device so that the musician feels the oscillation. The result will improve a player's perception of when the oscillator speaks and how it speaks. Controllers that communicate to the sense of touch can also incorporate kinesthetic forces [6] [7]. Good tools lend themselves to skillful operation - future work aims at affording better control to performers of synthetic electronic musical oscillators. References [1] Rt. Verrillo. "Vibration Sensation in Humans," Music Perception, Vol. 9, No. 3, pp. 281-302: 1992 [2] C. Sherrick and 11. Cholewiak. "Cutaneous Sensitivity," in Handbook of Perception and Human Performance, Vol I, pp. 12-1 12-58: 1986 [3] Jl. Bolanowskid, et al. "Mechanical aspects of touch," JiASA, Vol. 88, No. 5, pp. 1680-1694: 1988 ICMC Proceedings 1993 79 8A.4