Page  00000017 Time Slices, Graphic Scores and Music Composition Abstract This paper illustrates a technique for creating and realizing graphical scores from time-based visual media such as video. The score is a 'time-slice' from the video material. When the time-slice is sonified the audio will automatically correlate to the video. Many compositional decisions must still be made in constructing the sonic maps. Application of this technique in the composition of a sound score for a short abstract animation is demonstrated. How this process, and mapping in general, can be thought of as the making of metaphor, and how this is (or is not) significant in the making of art and music is briefly discussed. 1 Introduction Human experience is data mapping. It is that simple and that complex. Sound, for example, entering the ear as fluctuating air pressure, vibrates the eardrum which passes the signal to the cochlea. There it is finally transduced and coded into electro-chemical spikes of energy in the brain. Light is mapped similarly. Electromagnetic energy is processed by the retina and also finally converted into electro-chemical spikes in the brain. In the sensorium, sound and light exist as the same format of data. (How or why this data becomes the human experience we call "light" and "sound" is still a bit of a mystery, but we are at least starting to understand the mechanics.) Like the brain, digital technology stores information in a single format. In digital of course it's all bits. Audio and image are abstracted to binary numbers. At the level of number, sound and light are the same and so easily mapped into each other. For digital artists and composers some of the fun is in finding interesting ways to create these maps. And sometimes we can abstract yet another level and map the maps. Converting visual time-slices to sonic maps is one such process. 2 Time-slices A time-slice is created from time-based visual material such as a video. The time-slice is a static image built by extracting a single line from each frame of the, video and stacking these lines sequentially. Figure 1 shows a representation of video material as a series of Figure 1: Time slice through a scanline of a series of frames. rectangular frames with time represented as depth. The time-slice is constructed as a 2D cut running parallel through the time axis. Concatenating these individual scanlines onto a single image, one after the other, shows a picture of the evolution of the video on the horizontal line under investigation. With time represented on the x-axis, the resulting image is in a visual format that is similar to the visual representation of dynamic audio spectra often called a sonograph. 3 Sonic Maps Sonographs, constructed through Fourier analysis, can provide a visual representation of musical sound. (Cogan 1984) From the analysis a visual mapping is created, with time on the x-axis, audio frequency on the y-axis and brightness or color illustrating amplitude of energy in the audio spectrum. Figure 2 is a sonograph of the spoken word "Louisiana". It is a visual map of the sound. We can also invert the Fourier transform-we can reconstitute the sound from the visual analysis-map the map from the image back into its original sonic form. Essentially inverse Fourier analysis (sometimes we call it synthesis) takes an image and maps it to sound. This process provides a general way to make any image a sound. A time-slice for instance can be read as a sonograph and we could hear it. With some compositional technique we could also make it musical. (The digital synthesis program Metasynth by Eric Wenger and Ulsoftware, used for all the audio work discussed here, is designed specifically to read an image as a sonograph and to render a variety of sonic maps from that image.) 17

Page  00000018 I--ou---i--------s--i------a------na----- Figure 2: "Louisana" The compositional recipe is simple. Starting with time-based visual material (the example illustrated is an abstract animation, but it could just as easily be any video material), construct a time-slice. Now process the time-slice as a sonograph or graphic score (a sonic map), and convert it into sound-the image is sonified. (Another way of saying this is-compose/perform a piece of music using the time-slice as the score.) The compositional approach used here is digital synthesis, using the time-slice as source material. If the sonification is synthesized with a duration equal to the length of the video source of the time-slice, we create a coherent sound score that can accompany that video source. While there are several degrees of interpretative freedom within the process, the score will always correlate to the visuals. (Evans 1987) 4 meled (an illustration) Figure 3 shows a time-slice from an abstract animation titled meled. This slice was from the center scanline of the animation frames. Several techniques were applied to create a variety of sonic maps from the same time-slice. First the brightness values of the slice were inverted and edges were emphasized to bring a linear quality to the sound score-to thin the spectral density. (Like a sonograph, pitch, frequency and spectrum map to the y-axis.) As brightness maps to amplitude this thinning was used to avoid a dense spectral wall of sound. See Figure 4. Image resolution was reduced while duration for the map was unchanged, thus quantizing the image in time and in frequency. This created a piano roll version of the slice that was appropriate for a consonant and rhythmic sound space. See Figure 5. This image was mapped to a digital instrument that was pitched and percussive, constrained to a scale in Pythagorean tuning. A variety of sonic spaces are possible in the mapping process. (Evans 1989) In meled, for example, a different map was made with the same slice image using non-pitched percussion instruments of drums and shakers. A third track was created using a map blurred on the x-axis creating long sustained notes to be played by a zither-like instrument. Making sonic maps requires decisions regarding how the numbers will manifest as sound. These are compositional decisions. Often a high resolution image has to be substantially reduced in detail to accommodate musical space. This is most evident in a pitch space mapped to the y axis. For example mapping an image to just the white notes on a piano reduces the twenty kilohertz audio spectrum to a y-resolution of just fifty-two pitches. For the creation of a sound score to accompany an animation this is in some ways a feature rather than a problem. It ensures that the score correlates with the animation as it unfolds, but it does not create a cartoon-like one-to-one mapping of sound with image. (This is often called the "Mickey Mouse" effect in Hollywood.) Various image processing techniques are handy in massaging the time-slice before sonifying. The program Adobe Photoshop proves useful as a tool for music composition. For meled a few mappings were created and recorded as individual audio tracks. For each track the image was mapped to a unique set of sound samples. Each track is a true mapping of the time-slice, yet each has its own quality. Tracks were mixed to create the final score. When combined, the audio tracks build a heterophonic texture and a coherent piece of music. Image and sound also cohere when played together, as the sound is a mapping of the visuals, and the animation is a pre-compositional mapping of the sound. Inside the brain they are both still just electro-chemical spikes, but now they are integrated spikes. They contextualize each other--a metaphor is expressed, image and sound connect. 5 A composer's thoughts We are order-seeking creatures. Through our senses we receive signals from the world around us. Our senses convert them to electro-chemical data we then interpret, thus creating all of our experience. We try to make sense of these signals. What endures? What repeats? What changes? We look for structure, based on stored patterns of past experience and new patterns we receive through our senses. Recognizing and comparing patterns helps us to understand the sensory data and provides a basis upon which to make 18

Page  00000019 choices. Living is the making of choices based upon the receiving of signals and the perceiving of pattern within those signals. (Hawkins 2004) We compare new patterns we receive with patterns we already know. We try to map new information to old. We try to make connections and links within our neural networks--the making of perceptual and cognitive metaphors is the construction of knowledge. Artists are order-making creatures. Sometimes we want to simply understand what order is. What establishes order? What defines pattern? Digital artists deal with number as their medium. As we create an experience it exists as numbers and we can map those numbers freely-we can make new metaphors. If we use the numbers to create aural experience we make a sonic map. As a digital image is also nothing more that numbers it follows that we can create a sonic map of that image and hear it. Monet's water lilies can be a symphony, Beethoven's Ode to Joy a wash of pigment on canvas. Time-slices provide us a new way to see time and then a new way to sonify the temporal image. New maps. New metaphors. In making maps aesthetic decisions are made. What colors to use? What pitches? How will time flow. How will space be framed. The artist chooses. As a composer I spent years making choices while sheltered in the safety of formalism. With a loose Pythagorean base, I enjoyed the stability of mathematics, Da Vinci's mystical proportions, Kepler's "music of the spheres." It provided a protected harbor to berth my artistic ideas. From my academic training as a composer this was a comfortable place. I was secure in the surety of form. I spent years mastering the intricacies of Western harmonic practice, the fugue, the sonata. I marvelled at the power of repetition and variation, and the necessity of a well-structured utterance. Composition, in the traditional sense, is a stable and safe place to work. But then I entertained other possibilities. John Cage describes a three-tiered approach to considering composition. (Cage 1984) First is structure - the division of parts. Second is method-- the note to note procedure. And thirdly materials-the sounds (and silences) of the work. For Cage (at least in his earlier work) structure is determined by strict divisions of time, method is improvisation (free from any formalist baggage) and the materials are "chosen as one chooses shells while walking along the beach." Cage invented a process to create simple events. The listener, in seeking order, makes it music. Upon hearing, the listener creates the relationships of materials (the form). Form requires participation. Musical experience is interactive. It's no longer about the composer. It's listener focused. I find this idea poetic in concept and handy for art making. I am a digital artist. My medium, my "material" is number and number maps freely. (Mathematics, after all, is the ultimate metaphoric language.) These days my compositions (sound and image) start with a simple structure, a duration- two minutes. Sometimes I create a time-slice and ensure a correlation, a coherence of materials from the visual to the aural. Sometimes I don't. Instead I improvise a visual mapping of number that creates an abstract animation lasting two minutes. I compose an unrelated sound event of two minutes duration. I combine them. Now the visual and auditory channels in the cortex of the brain are engaged. We hear and see and make connections. We seek order. Events are synchronized by whim, order is perceived by coincidence. It is synchronicity. Just as in life, we constantly make sense of the chaos around us. It seems like order is always there just waiting for us to see it, to hear it, to construct it. (Evans 2005) Usually the old formalist in me won't leave it there though. I take a time-slice and create a simple sonic map and slip the new sound onto one track of the audio design. It's subtle, perhaps a bit cowardly, but it's enough to lock the experience. Pattern flows. The viewer/listener can make deeper sense (maybe), and perhaps extract some meaning. They hear the colors. They listen with their eyes. 6 References CAGE, JOHN, Silence, 1984, Wesleyan University Press: Middletown. COGAN, ROBERT, New Images of Musical Sound, 1984, Harvard University Press: Cambridge. EVANS, BRIAN, "Chance and a Hidden Order," The YLEM Journal: Computers and Music Special Edition, Vol. 25-no. 6 & 8, 2005, ed. Loren Means, YLEM: San Francisco, CA EVANS, BRIAN, "Enhancing ScientificAnimation with Sonic Maps," Proceedings of the 1989 International Computer Music Conference, CMA: Columbus. EVANS, BRIAN, "Integration of Music and Graphics Through Algorithmic Congruence," Proceedings of the 1987 International Computer Music Conference, CMA: San Francisco. HAWKINS, JEFF, On Intelligence, 2004, Times Books: New York. 19

Page  00000020 Figure 3: meled time slice through center scanline 'IL Figure 4: meled time slice with edge emphasis. (Value is inverted for illustration.) Figure 5: meled time slice quantized in time and frequency making a piano roll graphical score. (Value is inverted for illustration.) 20