Page  00000308 USING A 3D MODELLING PROGRAM AS A MUSICAL CONTROLLER Rodney Berry Communications and New Media Programme National University of Singapore Blk AS6, #03-41, 11 Law Link, Singapore 117570 rberry @ gmail.com ABSTRACT Real-time 3D computer graphics offer interesting metaphors for representation and control of computergenerated sound. Such metaphors can reach beyond the interface-building tools included in popular computer music synthesis programs. By exploring some simple projects using the Touch Designer modeling and animation environment, the author intends to expose some of the sound control possibilities afforded by this program and encourage further experimentation by others. 1. INTRODUCTION The main purpose of this paper is to document some preliminary attempts to use a commercial 3D graphics program for musical control. As such, it does not describe a finished system or seek to evaluate it in terms of usability. Instead the paper focuses on things found during the undirected, exploratory phase of research with the view that others may be motivated to make similar explorations of their own. Modern sound synthesis tools such as Max and Pure Data have recently extended their scope to include live performance of 2D and 3D graphics and the design of on-screen graphical user interfaces. However, such programs are still not quite capable of the visual richness that one can achieve with dedicated graphical tools, especially in the area of live 3D. This is why I chose a more visually-oriented program called Touch Designer (hereafter referred to as Touch) with which to look for interesting metaphors for the representation and control of computer-generated sound. 2. TOUCH DESIGNER Touch is a 3D modeling package that emerged as a realtime adaptation of Side Effects Software's high-end Houdini modeling and animation environment. Houdini is commonly used in cinema for procedural animation of particles, smoke and flying debris, as well as for geometry, textures and characters. Touch is similar, but aimed directly at the production of live real-time visuals and is mainly used by VJs and for interactive 3D art works. Like Houdini, Touch uses a procedural data-flow paradigm where each step in the operational chain continues to 'cook' and contribute to the final result. In this way, it has more in common with the 'join-the boxes' environments, popular among computer musicians (Max/MSP, Pure Data, Reaktor etc.), than it has with other high-end modeling and animation packages like Maya and 3D Studio Max. Inspired by early analog modular sound synthesizers, projects made using Touch are called 'synths'. The interface is organized into three main operational domains, that of Surface Operators (SOPs) for geometry, Channel Operators (COPs) for control, and Composite Operators (COPs) for textures (COPs are not covered here, but also show great potential for music making). 2.1. SOPs and CHOPs Figure 1. A torus shape is, in turn, flattened, stretched and twisted by a series of operators. To build a 3D object, the user begins in the domain of Surface Operators or SOPs, by choosing from a menu of generators to produce a basic geometry for modification. This generator could be a simple geometric shape, an imported file, or something more exotic such as an Lsystem, a Metaball, or an Isosurface. By connecting a generator to one or more filters, the initial geometry can be transformed through a variety of manipulations. In Figure 1, a Torus operator is fed to a Transform, then a Twist operator. Each operator adds its own modifications to the preceding one, and the cumulative result appears in the viewer. If we replace the torus with a sphere, the changes propagate down the chain of operators causing the sphere become flattened, stretched and twisted, just as the torus was. Channel Operations (CHOPs) create and manipulate control data. This data can then be sent to and from the other operators to allow live transformation of objects and images. Generators include live MIDI input, OSC 308

Page  00000309 messages and other data via network sockets, joystick, work by Story [5] and by Lyons [3] was done using mouse and tablet data, various oscillators and noise Houdini, predecessor of Touch Designer. generators, data from on-screen widgets (sliders, knobs The project described here is intended as a simple etc.), tables of data from files (motion capture, midi, and visually interesting controller for sound using the audio etc.) and even a clock with sun and moon position global positions of a set of points in an imaginary 3D at a given latitude and longitude. Filters include delays, space to control a set of sound parameters for making EQ, time stretching, triggers and envelopes, math music. Furthermore, I wanted to find a way to do such operations, sample and hold etc. things without relying on low level programming used in much of the earlier work mentioned. Ideally, it should also be possible to look at the visuals and listen to the |gg| sound and be able to understand something of the relationship between the two. This is why I opted for a |. simple sine wave additive synthesis model, the better to S:::see and hear the relationship between the images and,il the sound. Figure 2. CHOP network with graph of output showing simple ramp acquiring a quivering effect when the Spring operator is added. In the example above, a Timing ramp is added to the network editing space. We want it to ramp up to a value of two-hundred, then ramp up again from zero. The default range of the ramp is from zero to one, so we add a Math operator to multiply its value by two-hundred. We then add a Spring operator to make a quivering effect when the value returns to zero (Figure 2). It is now possible to export the output of this network to control the amount of twist in our earlier torus example (Figure 3) so that the torus shape is periodically twisted before it snaps back to its original shape with a springy motion. 3.1. Sound Component A bank of 30 oscillators was made using the Pure Data [4] synthesis environment (henceforth referred to as PD). The oscillators were tuned to the first 30 partials of a harmonic series relative to the tuning of the first oscillator. Osdiators!.-30 gI~ l~... " ****.........111.... 5>^^ ^ *^amon - ------- i ^^-------RCNT-s~^ ^ Figure 4. 30 sine wave oscillators controlled via MIDI by point position data from Touch. A MIDI continuous controller was assigned to the amplitude of each oscillator. Another controller was assigned to the panning of each oscillator's output from left to right. A third controller was assigned to offset the pitch of each individual oscillator in increments of 1/64th of a semitone. The outputs were then mixed before going to the main output. 3.2. Video Component Figure 3. Twist SOP's parameter box showing the strength parameter highlighted. 3. PARTICLES, POINTS AND PARTIALS Using particle systems to control music is certainly not a new idea. Cook and Tzanetakis [1] [7] used particles with physical acoustic models to simulate such real sounds as gravel and ice cubes etc. The works of Sturm [6], or Blackwell and Young [1] apply ideas from particle physics and other particle-based phenomena to control parameters for granular and other forms of synthesis. In addition, music and graphics synthesis Figure 5. Particle paths in controller space. 309

Page  00000310 To control the amplitude, panning and pitch-bend of each of the oscillators, a corresponding geometry synthesizer was built using Touch. To create a visible space that corresponds to the parameter space of the audio synthesizer, a cube was defined so that the x axis would correspond to pitch bend, y axis to left/right panning, and z to amplitude. A Geometry operator collects the x, y and z translations of all the points in the particle system, then a Lookup operator extracts these translations for each point and, as indexed by the input from the Constant operator, it generates 90 separate data channels. Figure 6. Particle system network The first controller was a particle system emitted from a central point in the space (Figure 6). An Add operator defines a point source for particles. This is fed to the 'particle source' input of a Particle operator. The number of particles is constrained so that there are always 30 particles at any one time. Once a particle dies, its point number is reused by a new particle being born. Each particle is given a point number to match one of the sine wave oscillators in PD. To move the particles around, a Metaball and a Force operator are connected to the force input of the Particle operator. A metaball is an implicit 3D surface often used to make objects that blob together like water droplets. Because it is defined as a force field with a set weight and radius, it is also used as a source of forces to act on a particle system. The Force operator is used to direct the forces according to its parameter settings as follows: * Radial Force pulls objects in toward the center of the system. * Direction x,y and z define a force vector along or around which the forces act. * Axial Force pushes objects along the axis defined by the force vector. * Vortex Force spins objects around the axis defined by the force vector. * Spiral Force pulls objects in toward the axis defined by the force vector. Other channel operators map these parameters to graphical sliders in Touch's control panel, so that they can be adjusted in real-time by the user (and also controlled with an external MIDI fader box). A slider is also added to control the strength of the Metaball's force field. When at maximum, the balls move very quickly and definitely in response to the applied forces. When in the negative range, all forces are reversed and the particles float lazily at the boundaries of the space. Figure 7. 90 channels ready to choke our MIDI buffer These are then fed to another network that scales the point position data into the 0 - 127 range required by the MIDI-Out operator. A Resample operator is also inserted to bring the data rate down from 60 samples per second to around 10 so as not to overload the MIDI connection. Admittedly, this would (and will) be better done using OSC messages but, for various reasons, is not practical at the time of writing. 3.3. Spring Model The continuous birth and death of particles can cause unwelcome discontinuities in the sound. Replacing the particles with Spring operators allowed a more consistent connection between the geometry and the sound. Having 30 separate Spring operators made it possible to fine tune the spring parameters of each individual point so they each have slightly different behaviors when in motion or at rest. To avoid the tedium of wiring up and setting 30 springs by hand, Touch Designer's scripting language (a simple but versatile language, similar in character to UNIX's C-Shell) was used to automatically construct and connect the various operators. In addition, a Logic operator was added to invoke a selection of scripts while the project is running. Script 1 makes the lower numbered points greater in mass than the higher ones. This results in the higher harmonics moving around the space more quickly and with less inertia. Script 2 turns off the mass calculation, so all the points behave as if they have no weight. Script 3 makes the higher numbered points heavier, for greater movement in the lower harmonics. Three buttons on the interface panel are used to switch between the 3 scripts. A further refinement might be to make a slider that smoothly changes the weighting from high to low harmonics. 310

Page  00000311 3.4. Grid Manipulations Figure 8. Grid: showing effect of Twisting and Noise on the spectrum To make something more repeatable than particles and springs, I placed a 6 x 5 grid in the space so that it would have exactly 30 points. It was then fed through various filters to distort its shape using an extra set of onscreen sliders (Figure 8). 3.5. Cosmetic Details For attractive display of information, each point was replaced with a small yellow sphere. The color of each sphere was set to make lower partials darker than the high ones. A transparent 2D spectrum display (Figure 8) shows the amplitude of each partial at the bottom of the display. This lets us better see the relationship between what we are hearing and what the balls are doing on the screen. In the Particle and Spring SOPs, a limit is set so the balls will not stray outside the space defined by the blue cube. When they strike the bounds of the cube, they bounce off. This causes interesting rapid changes in the sound when points are moving fast near the edges of the space. It is also a convenient way of limiting the output values to the 0 -127 MIDI value range. 4. CONCLUSIONS AND FURTHER THOUGHTS The system is surprisingly expressive. Even though this kind of use was not specifically intended by the authors of the software, it is clear that many capabilities necessary for live visuals run parallel to those conducive to music. In the case of Touch, these include a common time-base or clock, the concept of multiple parallel channels of information, and a very synth-inspired vocabulary of oscillators, filters etc. Probably the main thing that makes Touch interesting for musical control is the way nearly any parameter can be mapped to any other within the Touch environment. Because fundamentally, 2D, 3D and control data are all data arrays that can be converted, processed and converted some more, one quickly gets comfortable with this plasticity in thinking about mappings and metaphors. Admittedly, the visual interface consumes far more computing resources than the synthesizer it tries to control, but I think that this project hints at what might be done when a parameter space is visualized richly enough to let us get inside it. This project has ignited in me a desire for more tools that allow me to work with video and audio synaesthetically in the same workspace without one compromising the other. 5. REFERENCES [1] Blackwell,T., Young, M 2004 "Swarm Granulator" proceedings of 2nd European Workshop on Evolutionary Music and Art, Coimbra, Portugal 2004 in Springer LNCS 3005 pp397-406 http://doc.gold.ac.uk/-mas01tb/papers/SwarmG ranulator.pdf [2] Cook, P. R. "Physically informed sonic modeling (phism): Synthesis of percussive sounds". Computer Music J., 21(3):38-49. 1997 [3] Lyons, Andrew D._"Time Space Texture: An Approach to Audio Visual Composition"._Unpublished PhD Dissertation. University of Sydney, Australia, 2003. http://www.users.bigpond.com/tstex/Time_Spa ce_Texture.pdf [4] Puckette, M.. "Pure Data." Proceedings, International Computer Music Conference. International Computer Music Association, pp. 269-272. San Francisco, USA:1996 http://crca.ucsd.edu/-msp/Publications/icmc96. ps http://crca.ucsd.edu/-msp/software.html [5] Story, M., "Houdini and CSound" http://www.digitalcinemaarts.com/dev/csound/i ndex.html [6] Sturm, Bob L. "Composing for an Ensemble of Atoms: The Metamorphosis of Scientific Experiment into Music." Organised Sound, Vol 6. No 2, Cambridge, UK. 2001. http://www.composerscientist.com/content/rese arch/2001_OS.pdf [7] Tzanetakis, G. and Cook,P., "3D graphics tools for sound collections," in Proc. COSTG6 Conference on Digital Audio Effects, DAFX, Verona, Italy 2000. http://www.cs.princeton.edu/-gtzan/papers/daf x2000.pdf 311