Page  00000172 Description of an interactive, sound making, online computer music tutorial web book. Available for any browser on a Windows or Apple platform. Phil Burk, Larry Polansky, Dartmouth College douglas repetto, Columbia University Daniel Rockmore, Dartmouth College Mary Lee Roberts, Princeton University Most of the people in our group work as educators. And we have noticed that the young people we meet are anxious to work with found sounds that they have digitized on their PCs. These are sounds that they have gathered out in the natural world: geese on the pond down the road, rain on the roof above our studios, a powerful wind blowing down from the cold winter peaks. Just as we might have been fascinated, as youngsters, with the fireflies that we captured in a jar, and kept there for close examination, we find that our students are very much interested in capturing a sound on their computers, and developing analytical tools to examine the sound, then going on to a creative use of their found sound. The natural world hardly ends at the computer as far as our educational endeavors go. We hear our digitized sounds as extensions of the natural world. We too would like to know what it means to capture the sound of frogs and crickets, much like Pauline Oliveros did in the 1960s for her piece Alien Bog, take this captured sound and digitize it, and take our knowledge of mathematics and programming and perhaps cross synthesize one natural world with another. When our group gathers together we have the common thread of a need to understand the mathematics behind the world of sound: what can mathematics tell us about the sound world around us, where can we begin in modeling the natural sound world, what can we do as educators to try and explain the sound world that our students encounter everyday? Now is an especially good time to educate young people about the mathematics behind acoustics, especially since many of our students are so interested in capturing the natural sound world on their computers and using their found sounds to either model or modulate the natural world. We took our cue from our students and we decided to create an online computer music tutorial web book accessible via a browser. This short paper/demo describes our interactive online web book that combines tutorials in mathematical topics with direct and practical applications for a deeper understanding of computer music, synthesis, digital signal processing and computer based music composition. Our finished online book is a 5 chapter web site peppered with Java applets that demonstrate important and key aspects of mathematical applications for sound synthesis. From the FFT, to PVC, to number theory, every example is geared toward encouraging the user to experiment in sound and explore the mathematical basics for understanding sound right in the browser. Hearing Frequency and Amplitude The best way to understand frequency and amplitude is to hear the effect of changng theman We will control a sne wave oscillator wtach produces a very smooth tone The output of the oscillator wall be sant to an on-screen osulloscope so that we can see the effects visually The waveform display has time on the horizontal axis and the value of the oscillator output on the vertical axis It is updated automatically about once per second To D.o:. Move the amplitude scrollbar until you hear sound * Move the frequency scrollbar SListen to the effect of chanig each Notice how the wavefonn changes on the osulloscope. Try to play a simple tune using the frequency scrollbar STurn the frequency down untl you can't hear it. about I-20 Hertz Can you feel your speaker vbratng slowly with your hand At your own nsk. remove the cloth cover from your speaker if it comes off easily Can you see the large woofer vibrating? Ths Applet requires the JSyn plugon If the Applet fails to run, you can download the free JSyn plugun from here Fig.l Our applet that discusses the relationship between amplitude and frequency. We have geared this book towards young people who may be interested in sound synthesis but may not have an extensive background in mathematics and little access to sophisticated and expensive hardware and software. Our target audience is either advanced high school students or introductory college classes. With our web book students can get started making interesting sound experiments immediately, by using 172

Page  00000173 the Java applets, right in the browser. All of our applets are based on the JSyn audio synthesis API for Java developed by Phil Burk, a member of our team. The JSyn audio synthesis API has been used by our team to develop Java Applets that control real-time software synthesizers, this in turn has allowed for us to insert interactive compositions, or virtual instruments, into our web book. We were particularly interested in creating an introduction to computers and music that did not depend on some pre-existing language or application like Csound, SuperCollider, or Max/MSP. With the idea that mathematics is the building blocks for a thorough knowledge of the sound world around us, our approach starts with an explanation of the basics of sound and how we can understand mathematical functions and their applications for sound making and sound appreciation. All through the beginning sections of our web book we have many Java applets: one applet demonstrates what a sine wave sounds like, another applet demonstrates the basics of amplitude and frequency. Other extratextual learning devices include numerous sound files, animations, and graphics that apply directly to the explanations in the text. Never is a topic covered without some sound making mechanism, either an applet or a representative sound file, to make sure that the focus on mathematical realizations in sound stays in the center of the reader/listener 's consciousness. Chapter 1 provides a basic introduction to amplitude and frequency. The site then moves onto a discussion of timbre and what timbre can mean to an electronic musician who has a working knowledge of mathematics. For example, we have an applet that lets you draw and listen to periodic waveforms. We ask the question: "How much does the waveform (which is a result of its steady-state spectra) influence what you hear? Some waveforms sound brighter than others, some duller..." and the user can experiment with creating the waveforms and listening to them with our applet. We finish out our timbre discussion by raising the question:" what really makes instruments sound like they should, and how can we model them with our computers?" Within this topic we take some basic sound sources like trombones the voice, and violin and we remove the attack characteristics. We then challenge the reader/listener to take into consideration the basics of attack/decay/sustain within the context of timbre. How can we learn to model and build our own instruments from what we know about the natural world? Where can we find timbral inspiration from the natural world? We also discuss the importance of additive synthesis, sine waves, and psycho acoustic models here. Chapter 2 is a discussion of the digital representation of sound; how computers make and store sound, how computers do the math so that sound making applications are available tor us to use as composers. Here we discuss the basics of digital conversion for sound, data storage,data compression, the graphical representation of sound, and an introduction to digital manipulation of sound. The mathematics in this section are discussed within the context of how computers work:the importance of recognizing basic facts such as the Nyquist frequency, aliasing, and binary numbers. One feature that gives the viewer/listener a valuable hands-on experience is the discussion and representative sound files on bit width. It 's here that we explain and play sound files sampled at all kinds of bit widths, way down to an 8-bit file sampled at 5512.5 Hz. We have fun with our discussion on digital copying and we provide a nice listening break with a representative composition by composer David Mahler that illustrates, in a funny way, how a good and solid understanding of the math behind digital sound manipulation can lead to solid musical results. We also touch on the topic of representation in musical sampling: how can we take what appears to be a natural sounding speech pattern and manipulate it in some way (with our new found mathematical knowledge) so that the nature of the voice and speech understanding is completely masked by the synthetic context in which it takes place. Much of the heavy mathematical material comes with our sections on the frequency domain and the basics of DSP in Chapter 3. We begin with a discussion of the Frequency/Amplitude/Time Plot, and then move directly to an in-depth discussion of phasors and what the mathematical and compositional aspects of phasors can mean to musicians. Heading into trigonometry we supply the basic functions and the basic sine and cosine graphs to provide a clear understanding of what is going on when with basic waveforms. Sampling and Fourier Expansion is covered and then a long discussion of Fourier and the Sum of Sines is capped off with an applet that lets the user edit real or imaginary values in the Time Domain or the Frequency Domain. As the user edits the values in one domain, the other domain is automatically recalculated using an FFT. The magnitude and phase of the Frequency Domain is also shown with the applet. 173

Page  00000174 Play with anll FFT This Applet lesyou edit real or iaginary values in the Time Dmana or the Frequency Domain As you edit the values one domain, the other doman is automatically recalculated usng an FFT. The ma artude and phase of the Frequency Domai 1 is also shown. (Ve would like to thank Dave He for postig FFTLab to the web It as the starting point for developing thls Applet) ToDo S Click the 'Clear" button to zero out all the data. "Drag the mouse stralgllt aclos the top-left box labelled "Time Domai", "R,"9 NNot ce that a peak appears in the first bn of the Frequency Doman This is becuse a constant value has a frequency of zero*fundamental Draw asngle cycle of a,ne *ave in the same box Notice that the peak now appears in the second bin representing the 1* fundamental frequecy ofthe FFT SDraw two ne t the same box and notice the Frequency Dotan peak moves to the third bin representng 2* fandamental " Hit Clear and then dick once in the box so tt you et a ngle peak Notce that the Frequency Domain now how a constant Mantude Ths is because an "impulse' peak has a spectrum that contains all freque es equally F Hit Clear and then click on the first Im You ould ee a mne wave in the Frequetcy Domain This shows that the Time Dosno " Poke aromd in the Frequency Domain boxes, drawing vanous ngnals and ltcliling the results, until everything suddenly becomes Recalc I magi nary Real Itmagi no ry D s oagnitude n Pase Fig.2 FFT applet The DFT, FFT and IFFT and their inherent problems with frequency/time accuracy come next and the chapter ends with a survey of alternatives for the FFT and what a musician with the basics of mathematical knowledge can do to develop their own systems for composition. This chapter can be difficult reading for some students who are not comfortable with calculus. Mathematician Dan Rockmore, a member of our team, has written a great deal of this explanation. Dan is an expert on the FFT and it's numerous applications. Many of our educator friends have commented on the clear and concise language that Dan uses here. We like to think that we not only have a good explanation of the FFT for music makers, but we have a fun context to put it in: we make sure that our applets are not only educative, but they demonstrate the math in a musical context. Sampling and Fourier Epaqsalon The decoinFosltlon of a complex wavefornm into fts component phasors (,hich is pretty much the same as saying the denmnposition of an acoustic wavefoln into its component partials) is calle J F,:oruer ezpanlosn computer 9 hese sa99ple are then converted (us9ng what Is called a Fast Foirger "9a9sf9u, or 9 y D into what are called Fourer Fig.3 An example of some of our more whimsical graphics. Chapter 4 provides a survey of sound synthesis methods. Right away we offer two applets:the first applet gives the user the opportunity to add sine waves together at different amplitudes, the next example moves toward a more complex model with an applet that allows for control over spectral envelopes. An interesting subsection of this discussion goes into detail about Shepard Tones; here is one of the places where we discuss the importance of psychoacoustical studies in electronic music. This section culminates with an historical musical example of: For Ann (rising), by James Tenney. Filters and filter theory are the next topics we cover with an indepth discussion on filter equations, the FIR and IIR filter types and what these filters can do for composers. The Four Basic Types of Filters stopband stopband stopband Fig.4 A sample filter diagram from Chapter 4 Logically a discussion of Formant Synthesis is next with a representative applet for the user to experiment with. It is here that we offer up the question of how far we can go with creating "natural" sounding sounds. Can we really model speech successfully? If so, how can we go about doing this? Earlier in Chapter 4 we had presented a series of sine wave generated speech examples loaned to us from the Haskins Laboratories at Yale University. In the later part of Chapter 4 we present the question of how important it might be for a composer to want to recreate natural sounds with synthetic means. Examples from Paul Lansky's work are used to introduce the user to music that has a basis in natural sounding speech patterns, yet the overall sound is synthetic. An applet for creating frequency-modulated sounds begins our discussion on modulation. We provide an assignment for experimenting with FM: the student is instructed to take a LFO controller built into the applet and with the usual parameters, experiment with setting modulation rates, depths, and frequencies. We also take the time to discuss some aesthetic concerns about making music with oscillators. Many of our students are familiar with sampling; most of them, however, are just now experimenting with digital oscillators and modulation techniques. 174

Page  00000175 Low Frequency Modulation In this oample, we are using a ne wave oscillator to modulate (wiggle) the frequency ofa triangle ave oscillator The SineOsallator frequency is st to a very lowvalue, like 5-10 Hz so that it anuses a slow wggle in the TnangleOsallator frequency The SmeOsallator in this situation is referred to as an LFO, or "Low Frequency Oscillator' The one ave is added to a constant value that determines the center frequency of the Tnrngledsllator. ToDo * Increase the Modulation Depth" fdad Notice that it causes the frequency ggles to get biger This contrls the amplitude of the SineOamllator " Increase the "Mdulati on Frequency" fader Notce that it causes the frequency wggles to get fater This is sometmes called "Modulation Rate" This controls the frequency ofthe SineOsallator * Set the Modulation Depth to 00 and t ovthen move the "Center Frequency" fadttr Notice that the sine oscillator goes up and down in pitch without any modulation * Set the ModDepth and the ModFreq faders to ther ioaMmut values Set the "Center Frequency" to about 150 0 Notice how when the output of freqAdder goes low, the frequency ofthe tnOsc is at its lowest Thtis Applet requires the JSyn plugn If the Applt fails to run. you can download the free JSyn plugin from here Modulation Frequency 1ý-j 0 added bonus of a historical tribute to the late composer Herbert Brimn. There are applets to make music with samples and an applet that explains and offers a drum machine for composing. Fig.5 Low Frequency Modulation applet Op*0 00 1 0 2 [ deoen2 71q4 freqMult depth amplitude (dB) o* Oo i 2 detansa7191 freqMult depthdex 03l a Hear Sust op* 0 03o 2 1 f freqMt depth index We cover all the usual suspects with sound making applets for demos: FM, waveshaping, and we end the chapter with a discussion of Granular Synthesis and Physical Modeling. The Physical Modeling chapter is especially fun for students to spend time with. A Karplus-Strong applet gets them started right away, then we head on to moreu interesting models: Charles Sullivan's model of a feedback-able electric guitar, Perry Cook's model of the human voice: SPASM, and we end the chapter by offering a section on: "Understand the Building Blocks of Sound" focusing on the importance of analysis of the basics of natural sound. We explain why it is important that we know what acoustical phenomena happens when we pick a guitar string, for example, and how the analysis data gleaned from finding out what happens when we pick a string can be used in numerous musical contexts designed on a digital system. Chapter 5 begins with a survey on: "Introduction to the Transformation of Sound by Computer ". Some of this territory may already be familiar to some students, but we include some listening example gems in order to provide a historical context. Tape music/Cut and Paste, Time Domain Restructuring, and Sampling are discussed. This section has the Fig.7 Drum Machine Applet Once the composer has the building blocks of sound under her belt we offer some sections on signal processing: Reverb (with a very cool multi-tap delay applet and a guitar stomp box style applet), and some unusual sound file examples made by douglas repetto. There is a section on Localization/Spatialization where we start with a filter based localization applet. This applet lets you pan, fade, and simulate binaural listening (best if listened to with headphones). The question of what we think we hear and what we may actually be hearing is raised, and a discussion on listening aesthetics and biological concerns lend toward a better understanding of what it means to be a musical listener. Also of note here is our "Introduction to Spectral Manipulation" where we describe the phase vocoder. We have provided a number of listening examples to compliment the text where we outline the basics of PVC parameter tweaking. This section concludes with a composition offering by Steve Reich, where he discusses Phase Vocoding within a non-electroacoustic environment. Convolution comes next with a discussion of Cross Synthesis. We had already discussed convolution techniques in the context of reverb and room simulation earlier in this chapter, but it is here that we go over some aesthetical concerns regarding, for example, what we can think of "natural" sounding sounds, and what we can do with cross synthesis to blur the lines of "natural" and "unnatural" sounds. And naturally, what comes next is a section on Morphing with a discussion on Morphing the Spectral Centroid. Our book concludes with a section dedicated to a survey of interfaces that are designed to provide a graphic environment for manipulating sound. As with our discussion on Time Domain Restructuring earlier in this chapter,we have provided some historical examples of graphic interface technologies, particularly the work of Norman 175

Page  00000176 McLaren and Hugh LeCaine. We discuss the advantages if the UPIC system, the Hyperupic system, and the QT-coder in SoundHack and our own douglas repetto's application: squiggy, to name a few. All along in the section we provide plenty of applets to demonstrate the technologies in these systems and numerous listening examples from these applications and the composers who use them. Originally we had thought that our book would provide an inexpensive and easily obtainable sound making devise for people who are unable to purchase software and hardware. We had an image of a young person going to a library and with the simple use of headphones, she could learn all about sound synthesis while getting a good dose of the behind the sonic scenes math. Our sub-goals were to have extra bits of information on the history of computer music composition and technology, a basic education in acoustics and psychoacoustics, and lastly and certainly not least, a creative tool for a budding computer music composer. 176