Page  00000001 SPAA- AN AGENT BASED INTERACTIVE COMPOSITION Michael Spicer Singapore Polytechnic School of Info-Communications Technology ABSTRACT An interactive composition for flute and computer is presented, entitled SPAA. This piece uses an interactive composition environment that makes use of an intelligent agent to realise the computer part. This agent plays back samples of the live flute, with appropriate transformations, in order to achieve a predetermined musical structure. The agent decides which samples to play, and what signal processing to apply by analysing its' recent output combined with the live flute part. 1. INTRODUCTION SPAA 1 is the first piece in a suite of interactive compositions that feature the interactions between synthetic performers and an improvising flutist. (The flute was chosen, as I play flute, and it has a pure tone that I thought would be easy to analyze. It is obviously also possible to use other instruments). The synthetic performer is implemented as an autonomous agent written in C++. SPAA stands for Signal Processing Autonomous Agent. The design of this agent is loosely derived from the basic agent design used in two other goal driven agent based Interactive composition environments I have built, AALIVENET [1] and AALIVE [2]. The biggest difference is that instead of the agent controlling a MIDI synthesizer, as in the previous two systems, the agent in SPAA manipulates samples of the live flute performance. This is not a purely free form improvisation environment. The overall form of the piece is pre-determined, which provides a set of goal states to guide the agent in its decision making process. The live performer and agent performer combine to try and realize this form as closely as possible. The synthetic performer may cooperate (reinforce) or may interfere (contradict) with the live performer in order to achieve the goals as they change throughout the duration of the piece. My motivation for this piece was an interest in the interplay between the live flute player and the agent as they try to realize the form. 2. STRUCTURAL OUTLINE OF THE PIECE The structure of the piece is articulated as a set of goal states, indicating how various musical dimensions change over the course of the piece. These dimensions include: A ODverage pitch A ODverage note rate T ODimbre (amount of high frequencies present). D DAverage Loudness The states are stored in an array of C++ objects that encapsulate a four dimensional vector. The performance of the flute is be periodically analyzed, as is the output of the agent. An error is calculated, based on the values of these four dimensions. The agent then chooses which buffer to play so as to minimize the magnitude of this error. 3. AGENT DESIGN The agent is based on a sample playback system, which could be considered a descendent of the tape delay/ digital delay looping systems that have been commonly used over the last forty years. The structure of the agent is shown in Figure 1 below. Buffer0 Buffer 1 Buffer 2 Buffer N Goal States Figure 1. Overall structure of the SPAA agent. The agent has a number of C++ objects (currently ten, but the final version will have many more), each containing a buffer of audio input from the live flute performance, lasting about two seconds. The agent program, invoked by the PortAudio callback function, chooses which buffer to play, and how the audio in each buffer can be processed during playback. These decisions are made by comparing the combination of last live sample the agent has "heard" (analyzed), as well as its own output, with the current goal state. The agents' percepts are derived from analysis of the incoming audio stream. Each of the C++ objects containing the audio also has methods used for this

Page  00000002 analysis and attributes to store the results. These buffers are filled in the PortAudio callback function that runs in its own thread. The analysis is done asynchronously in the main thread. To do the analysis, the buffer is divided into a number of windows, each 1024 samples long. The R.M.S. and a F.F.T. of each window are calculated, and from these, a normalized measure of the average amplitude, average pitch, average amount of high frequencies present, and number of note attacks, can be calculated. These values are stored in a four dimensional vector that is representative of the flute performance stored in the buffer. The analysis result is stored in the same form (a four dimensional vector) as the goal states that are used to specify the desired evolution of the piece, so it is simple to calculate the current error by subtracting the sum of the analysis results from the most recent input and the currently playing buffer from the goal state. The buffer with the smallest magnitude error vector is chosen as the next playback buffer. One area that is currently being explored is simple real-time DSP techniques so as to reduce the errors in the timbral and average rate dimensions. Waveshaping and amplitude modulation/ring modulation can cheaply add more high frequency components. Low frequency amplitude modulation with suitably shaped ramp waveforms can create the effect of more note attacks. 4. IMPLEMENTATION The system is written in C++, and developed on a G4 Macintosh Powerbook, using Xcode. In order to make the program as portable as possible, I have used no platform specific API's. PortAudio is used to handle the audio I/O, OpenGL for the display, and Glut is used for handling the user interface (keyboard, mouse and menus). All of the analysis of the input buffers is done during the glut idle time function. I am using some global boolean variables to act as semaphores, so as to avoid the analysis and callback threads to interfere with each other. This is seems to be working ok. In order to get a clean signal, without a lot of spill, I am close micing the flute with a dynamic microphone. It is important to set the gain of the microphone correctly, so that the amplitude measurements corresponded appropriately with the system goal states. 5. FUTURE WORK AND CONCLUSION The current state of the software (June 2005) is at the "proof on concept" stage. All the parts are in place, and they work, but are quite crude. Much work needs to be done on the analysis stage, so as to extract higher level knowledge from the raw input signal. The agent function could be improved by adding some classification capability. The DSP routines used to enhance the sample playback are very rudimentary. Also, the user interface is not very friendly. Even so, the system as it stands is usable and shows the potential for using the agent approach for building interactive composition systems. 6. REFERENCES [1] Spicer, M.J. "AALIVENET: An agent based distributed interactive composition environment paper", Proceedings of the International Computer Music Conference, Miami, USA, 2004. [2] Spicer, M.J., Tan, B.T.G. and Tan, C.L "A Learning Agent Based Interactive Performance System." In Proceedings of the International Computer Music Conference, pp. 95-98. San Francisco: International Computer Music Association.2003