Page  481 ï~~The Sound of Fruits and Vegetables: Scientific Visualization with Auditory Tokens Gary S. Kendall Lonny Chu Center for Music Technology Northwestern University Evanston, IL 60208, keywords: visualization, audification, 3-D sound, psychoacoustics ABSTRACT: Fruits and vegetables provide a sound world with which every human being has some familiarity. That is why the sound of fruits and vegetables being struck, broken, torn, and smashed provides an ecologically valid basis for an auditory token library to be used in scientific visualization. Introduction. Scientific visualization is a catch word for a variety of techniques being developed to help scientists represent and conceptualize complex phenomena by means of advanced computing. The techniques that have received the most attention center around visual perception with three-dimensional computer graphics, although increasing attention is being given to the potential role of auditory perception with three-dimensional sound. Most applications of sound in scientific visualization have been somewhat ad hoc. While many forms of graphic representation are deeply embedded in societal communication and in our educational system, the auditory representation of information is conditioned largely by experience with film, television, and sounds from everyday life. Since there are no other common idioms or history of experience with using sound to communicate in this way, we are now at the point of inventing how auditory information will be used for data presentation. This situation should inform us both about the present role of auditory information within the culture and the need to establish a context for new applications with advanced technology. Sound is a particularly important source of information in everyday life, not just speech and music but also the sounds of everyday events. This suggests that the use of auditory information in scientific visualization is well served by taking advantage of everyone's base of experience with everyday sound, that is, that it take advantage of ecological validity (Neisser 1976). Token Library. Seeking an approach to visualization that is extendible to many contexts and within which idiomatic representations could be developed, we have created a library of auditory tokens that can be applied to many types of data representation. These tokens are short sound samples that have been carefully selected for data representation. Most attempts at sound synthesis for scientific visualization have mapped experimental data onto synthesis parameters. This has often proven to be ineffective because of the complex perceptual relationship between signal processing parameters and perception. The auditory token library attempts to remedy this problem by subjecting the tokens to perceptual scaling techniques that insure that they appear to be evenly spaced. The library catalogs the tokens along perceptually relevant dimensions, including azimuth and elevation for spatial presentation. Even though most single auditory tokens are short events, continuous, fused sound can be created by combining many tokens in a manner similar to granular synthesis. Adjusting the repetition rate enables the user to manipulate the relationship between the data and the thresholds for perceptual fusion and fission, thereby revealing different aspects of the data. The mixing of large numbers of tokens creates sound masses at a level of complexity comparable to complex events in the physical environment. The Selection of Tokens. The criteria involved in selecting the sound samples to be used as tokens required that the sounds be interesting to the listener, vary widely along numerous aural parameters, and possess strong ecological familiarity to the daily experiences of the listener. We considered the work of Freed and Martens (1986) whose study of the 'perceived hardness of mallet" was performed with kitchen pots. We decided that pitched sounds, even the pitched sound of kitchenware, created a potential conflict I C M C PROCEED I N G S 199548 481

Page  482 ï~~between musical structure and ecological considerations. We could use sounds with "pitch height", but not "pitch chroma" (Shepard 1964). After much deliberation, we decided that sounds produced through the manipulation of fruits and vegetables would provide a body of tokens perfectly matched to these needs. From the vegetables, we were able to generate many interesting sounds varying widely along many dimensions which were very pleasant and easy to listen to. Most importantly, since culinary experience is shared by people around the world, the vegetables resulted in a body of sound which is ecologically relevant to most people, allowing for ease of comprehension when used in visualization applications. Recording Process. We proceeded to collect our vegetables and record the sounds in order to build our auditory token library. In choosing our vegetables, we selected items which varied in their physical properties and which lent themselves to different forms of physical manipulation. For breakable vegetables such as carrots and celery, ecologically valid dimensions include the size of the vegetable and the force of the break. Okra produces particularly distinct breaking sounds because its internal construction is so different from its external shell. Lettuce and other leafy vegetables can be torn. One tends to interpret the length of the tear as related to the size of the leaf and the speed of the tear as related to effort. Soft fruit such as oranges and tomatoes can be struck against one another. The sound of the orange depends on the force of the contact. With extreme force, tomatoes cross over a threshold and break. Through experimentation, we successfully collected a large and diverse body of raw samples which we then edited and organized into a format compatible with our visualization software. Current Status of the Project. Our plan is to collect direct rating of perceptual properties along scales that are ecologically relevant. We have not yet begun the collection of perceptual judgments, but we have been able to sort out many of the dimensions of organization and to assemble a preliminary version of the token library. With this preliminary library, we used our visualization simulation software to create test examples of auditory visualization. In our test examples, each of the tokens is mapped to a specific datum. A data stream is then fed into the main software application which in turn uses the datum-to-token mapping along with timing information from the data stream to generate our test examples. The preliminary results suggest that vegetables and fruits represent a ripe area for research into auditory communication. The sound examples are rich in meaning and easy to digest. References Freed, D. J. and W. L. Martens. 1986. "Deriving Psychophysical Relations for Timbre." Proceedings of the 1986 International Computer Music Conference, Paul Berg Ed. Neisser, U. 1976. Cognition and Reality. San Francisco: Freeman. Shepard, R. N. 1964. "Circularity in judgments of relative pitch." Journal of the Acoustical Society of America 36: 2346-2353. 482 2IC MC P ROC EE D I N G S 1995