Raster Scanning: A New Approach to Image SoniU cation,
Sound Visualization, Sound Analysis And Synthesis
Woon Seung Yeo and Jonathan Berger
CCRMA, Department of Music, Stanford University
woony@ccrma.stanford.edu
Abstract
Raster scanning is a technique for generating or recording
a video image by means of a line-by-line sweep, tantamount
to a data mapping scheme between one and two dimensional
spaces. While this geometric structure has been widely used
on many data transmission and storage systems as well as
most video displaying and capturing devices, its application
to audio related research or art is rare.
In this paper; a data mapping mechanism of raster scanning
is proposed as a framework for both image sonil cation and
sound visualization. This mechanism is simple, and produces
compelling results when used for sonifying image texture and
visualizing sound timbre. In addition to its potential as a
cross modal representation, its complementary and analogous property can be applied sequentially to create a chain
of sonil cations and visualizations using digital 0 lters, thus
suggesting a useful creative method of audio processing.
Special attention is paid to the rastrogram - raster visualization of sound - as an intuitive visual interface to audio data.
In addition to being an efl cient means of sound representation that provides meaningful display of signil cant auditory
features, the rastrogram is applied to the area of sound analysis by visualizing characteristics of loop 0 lters used for a
Karplus-Strong model. Construction of new sound synthesis
systems based on texture analysis / synthesis of the rastrogram is also discussed.
1 Introduction
Data conversion between the visual and audio domain has
been an active area of scientil c research and various multimedia arts. Examples include waveforms, spectrograms, and
numerous audio visualization plug-ins, as well as visual composition and image sonil cation software such as Metasynth
(U&I Software) and Audiosculpt (IRCAM).
Since these conversions essentially represent data mappings, it is crucial to understand and utilize the nature of
Scan line
Return line
Figure 1: Raster scanning.
datasets in both audio and visual domains to design an effective mapping scheme. The temporal nature of a sound and
the time-independent, two-dimensional nature of an image
requires that data mappings between the two media address
these fundamental differences.
1.1 Raster Scanning as a Data Mapping
Raster scanning is a technique for generating or recording
the elements of a display image by sweeping the screen in
a line-by-line manner. More specil cally, it scans the whole
area, generally from left to right, while progressing from top
to bottom of the imaging sensor or the display monitor, as
shown in 0 gure 1.
In addition to being the core mechanism behind most video
display and capturing devices, geometric framework of raster
scanning - a mapping between one- and two-dimensional data
spaces - can be found in other places such as communication
and storage systems of two-dimensional datasets. This is, in
fact, the property that receives primary attention in our choice
of raster scanning as a new mapping framework between image and sound.
Raster scanning provides an intuitive, easy to understand
mapping scheme between one- and two-dimensional data spaces.
This simple, one-to-one mapping also makes itself a totally
reversible process: data converted into one representation can
be reconstructed without any loss of information.
34
0
