Page  360 ï~~Virtual Mutant Theremin Mark Bolas Phil Stone Fake Space Labs Sterling Software 935 Hamilton Ave Nasa Ames Research Center, MS 262-6 Menlo Park, CA 94025 Moffett Field, CA 94035 In 1989 Bolas and.Stone completed the Virtual Theremin (1) while working with the VIEW Lab at NASA Ames and the Stanford DesignProgram. The VIEW project (2), the first modem implementation of a virtual reality system, incorporated a tracked head-mounted display; gesture sensing glove; and what was, for the time, a high-speed rendering system. As audio capabilities were integrated into the system, it became apparent that the resources for a new musical instrument were in place. The Virtual Theremin and Virtual Drums Software was designed by Stone to facilitate the triggering and modulation of sound events on a MIDI-controlled synthesizer (an Ensoniq ESQ-m) in response to arbitrary actions generated by the VIEW system participant. An early goal of the VIEW project was to explore multi-sensory immersion: audio was recognized as a rich information display often neglected in traditional computer interfaces(3). In the VIEW system, audio was used to signal discrete events (e.g., the recognition of a hand gesture or the collision of two virtual objects) as well as to display continuous data (e.g., target proximity, contact pressure, etc.). The Virtual Theremin was originally designed as a shakedown of the VIEW audio subsystem; it tested the connection of VIEW data and events with audio cues. A simple "one fingered point" hand gesture triggered a continuous tone on the synthesizer and multiple pointing gestures would create many simultaneous tones. Closing the hand into a fist terminated all sounds. While the sound sustained, hand motion changed its character: forward motion modulated pitch: sideways motion modulated vibrato depth; and raising the hand increased loudness. Toward the goal of creating a multi-sensory instrument, Bolas created a graphical environment which corresponded to the audio and spatial structure of the Theremin and the player's actions. The overall instrument appeared as a box-shaped "nest" resting in the middle of six intersecting pillars. The nest served as the creation point of objects - a new object for each new sound. The pillars indicated the directions in which pitch, vibrato and loudness were changed. Playing the Virtual Theremin quickly became both an audio and a visual composition task. The ease of constructing and playing this configuration of the Virtual Theremin led to the development of several others. One variation enabled an arpeggiated chord to be triggered by a "two-fingered point" gesture. Hand movement in forward direction transposed the chord. The interactive nature of the Virtual. Theremin quickly led to interesting and often unplanned audio and visual combinations. 360

Page  361 ï~~For example, creating and terminating sustained tones while sweeping the hand forward created a crude Shepard tone that visually appeared much like a moving barber pole. Another interesting combination was formed when visual objects were designed with high contrast stripes. When these objects were in close proximity, and thus frequency, audio beats would be heard while Moire patterns were seen. Another instrument, the Virtual Drums, triggered percussive sounds when the hand intersected the six sides of the box-like "nest". Breaking the plane of each side produced a distinct drum-like sound. The Virtual Drums highlighted a problem when using the VIEW system as a musical instrument - it sounded like a drummer that couldn't keep time. Tracking and rendering delays created non-corresponding visual and audio lag which made playing the Virtual Drums an exercise in "musical prediction". This made rhythmic drumming quite difficult - although the effect was perversely entertaining. Conclusion The musicality of these initial forays was at best crude. Certainly, the degree of expressiveness attained by the Virtual Theremin was nowhere near that of its namesake. The Virtual Drums highlighted the inherent flaws of current tracking and rendering technology and their inability to keep up with musically-significant timing. However, the connection between gesture sound and sight was quite liberating and very enjoyable to experience. Moreover, the flexibility of design was nearly infinite, and this was most exciting. With adequate resources, in a virtual environment, musical instruments of fantastic capability could be constructed - responsive to any measurable gesture of forms inaccessible in the physical world and limited only by the imagination of the designer. 1) Trubitt, Rudy, Electronic Musician, July 1990, Vol. 6 No. 7 2) Fisher, S.S., Wenzel, E. M., Coler, C. & McGreevy, M.W. (1988) Virtual Interface Environment Workstations, Proc. Hum. Fac.Soc., 32, 91-95 3) Wenzel, E.M., Stone, P.K., Fisher, S.S. & Foster S.H., (1990) A System for Three-Dimensional Acoustic "visualazation" in a Virtual Environment Workstation. Proceedings of the IEEE Visualization '90 Conference, San Francisco, 329-3 37 361