ï~~Model-based interactive sound for an immersive virtual environment.
Robin Bargar*, Insook Choi*, Sumit Dast, Camille Goudeseune*
*Audio Development Group, National Center for Supercomputing Applications, University o f Illinois at Urbana-Champaign
tElectronic Visualization Lab, EECS, University of Illinois at Chicago
We discuss an audio rendering pipeline that provides real-time interactive sound synthesis for virtual environments. Sounds are
controlled by computational models including experimental scientific systems. We discuss composition protocols and software
architecture for hierarchical control and for synchronization with graphics. Rendering algorithms are presented for producing sound
from a physically-based simulation of a chaotic circuit and from higher-dimensional topological structures.
In this paper we discuss the implementation of a rendering
pipeline designed to bring sound synthesis and composition as
research components into virtual environments (VE). We find
that VE research provides a platform for projects closely
related to computer music composition. We also find the VE
research community is interested in the potential relevance of
composition for their work, and the relevance of their work
for composers. We have been developing a software-based
sound synthesis and composition protocol to enhance the
possibilities of collaboration. This protocol defines a pipeline
from computational models to sounds. Along this pipeline we
identify endeavors related to computer music including
real-time sound synthesis, gesture-based interaction,
composition algorithms, physically-based sound production
models, and techniques for synchronizing sound with
A rendering pipeline encourages composers to consider the
entire synthesis process as a composition. Generating and
controlling complexity is among the most difficult tasks of
computer music, and the pipeline model is valuable for
connecting complex systems with rendering engines. Rapid
low-level communication from complex model to sound
synthesis engine permit the composer to use interactive control
of the complex model to control the sound.
2. VE Systems
Virtual environments are multiple-engine computation
systems converging toward solutions for immersive human
interface. Immersion has been associated with two classes of
experience: fictional constructs and feedback constructs1.
Music listening may be identified primarily with the first class
and music performance primarily with the second. In order
that sound provide aspects of both classes of experience we
embark toward a familiar goal: techniques for generating
dynamical audio spectra informed by real-world analysis,
using efficient numerical encoding that allows synthesis with
interactive variation in real-time.
Existing computer music systems provide some of these
capabilities in specialized hardware. Rather than adopt existing
music systems we have focused on the importance of
demonstrating that sound synthesis is relevant for
general-purpose computing. We want researchers to have
immediate access to sound computation in the same language,
operating system and control flow that supports standard
computing and graphics rendering engines. Therefor our
pipeline is written in unix/c/c++ to maintain potential
portability and scalability and stay close to graphics
architectures and their user communities.
2.1 3D Primitives
The reality image in computer graphics is encoded in an
objective 3D embedding space. This objective encoding
describes the range of potential subjective views that may be
obtained from the image. Actual 3D rendering depends on
hardware that is downstream from the objective 3D encoding.
We adopt similar subsystems for audio. A primary software
language encodes oscillations, envelopes, and sound
propagation information, and manages rendering of dry
(non-localized) sources. Secondary, scalable display
subsystems are used to generate localization and depth
cueing. Regardless of rendering hardware, 3D features of the
propagation environment are encoded as attributes of the
primitive description of a sound.
2.2 The CAVE virtual environment
The primary testbed for our sound system is the CAVE, a
surround-sound, surround-screen, projection-based VR
Figure 1. CAVE Automatic Virtual Environment
University of Illinois
1. Fictional constructs involve an observer's "willing suspension of disbelief' that supports the diegesis (the narrative world created in literature, theater and cinema).
Feedback constructs are "everyday experiences" that an observer constructs by taking actions and observing their consequences through multiple sensory modalities.
ICMC Proceedings 1994