Page  81 ï~~The Continuum: A Continuous Music Keyboard Lippold Haken, Radi Abdullah, and Mark Smart CERL Sound Group Electrical and Computer Engineering University of Illinois, Urbana, Illinois 61801 ABSTRACT The Continuum is a new type of polyphonic music performance device that is now at the prototype stage. It is approximately the same size as a traditional music keyboard, but has a continuous playing surface instead of discrete keys. The Continuum tracks independent x, y, and z (pressure) position for up to 10 simultaneous notes. The playing surface is constructed using magnets suspended over Hall-effect sensors by a compressible medium. Performance skills required for the Continuum are quite different from those required for traditional music keyboards, due to the Continuum's sensitivity to the exact placement of fingers. 1. INTRODUCTION Existing MIDI keyboards provide the musician with key velocity and polyphonic aftertouch capabilities for controlling sound synthesis. Polyphonic aftertouch allows the performer continuous control over each individual note in a chord. These capabilities are extended by certain experimental keyboards, such as Moog's clavier (Moog 1982). Moog's clavier measures not only pressure aftertouch, but also other parameters including the exact horizontal and vertical location of each finger on its keyboard key. The Continuum's playing surface is quite different from Moog's clavier: instead of discrete music keyboard keys, the Continuum has a continuous performance surface. Figure 1 shows how the locations of fingers on the Continuum playing surface affect pitch, loudness, and timbre. The Continuum allows continuous control of vibrato (either in pitch, loudness, timbre, or a combination of these); continuous crescendi and glissandi; and any tuning system (much like a string instrument with a fretless fingerboard). The performer is able to bring out individual notes in a chord not only by increasing their loudness, but also by changing their timbre. LIGHT FINGER PRESSURE-QUIET HEAVY FINGER PRESSURE-LOUD HIGH PITCH OVER FINGERBOARD HIGH PITCH / / LOW PITCH OVER FINGERBOARD "~CONTINUUM" ~PERFORMANCE ~~SURFACE Figure 1 A timbral arrangement for the Continuum which could be used with violin sounds. In this example four notes are being played; the index finger is extended to cause a brighter timbre (synthesis of a violin being bowed closer to the bridge) for the highest note than for the other three notes. 81

Page  82 ï~~2. EVOLUTION OF TIE CONTINUUM The Continuum has evolved through many designs since the early 1980's. These designs are interesting because they all approach the same problem in very different ways. We now present these designs and discuss their advantages and disadvantages. 2.1 Photoelastic Material The original Continuum uses a photoelastic playing surface, as shown in Figure 2. The playing surface is lit by a single frequency polarized light source from the underside. When the performer presses a finger on the photoelastic, the resulting change in light polarization causes a ring pattern to appear around the finger. As the performer presses harder, the number of rings in the pattern increases. A CCD camera continually monitors the playing surface (for a full length 5-foot Continuum several cameras need to be used). The image from the camera is interpreted by a signal processor programmed with circle-finding algorithms; these algorithms find an x, y, and z (pressure) for each individual finger. 1/4 IN. PHOTOELASTIC ON 1/4 IN. GLASS RING PATTERN DEPENDS ON / SINGLE FREQUENCY PRESSURE LIGHT SOURCE CCD CAMERA Figure 2 The original design of the Continuum uses photoelastic material in the playing surface. This photoelastic approach results in an attractive playing surface, with interesting and continually changing ring patterns. Unfortunately, these ring patterns require an excessive amount of image processing for interpretation. If several close pitches are played the ring patterns overlap (as shown in Figure 2), and resolving the exact location and pressure of all the fingers in a note cluster is not simple. These computational problems, combined with the smell of melting plastic caused by the excessive heating of our light source, were the demise of this design. At the time we were working on this design a simpler method was developed at McGill University (Johnstone 1985); it avoided the excessive computations and used a dirferent type of plastic illuminated from the side. 2.2 Conductive Rubber Rubber is normally an insulator, but conductive rubber can be made by mixing carbon fibers into the rubber during the manufa cturing process. If the rubber is manufactured so that the carbon fibers are all oriented in one direction, the rubber is much more conductive in the direction of the fibers than in other directions. We designed a Continuum using a 0.5 inch thick rubber sheet with vertical carbon fibers, as shown in Figure 3. This rubber has very high horizontal impedance, but much lower vertical impedance. In our Continuum design, the vertical impedance is measured by detecting current through the rubber. As sh~own in Figure 3, a conductive sheet on the top side of the rubber 82

Page  83 ï~~supplies a constant voltage, and detecting pads on the bottom side (connected to a multiplexed A/D) measure the current. The rubber has some "at rest" impedance. When finger pressure is applied, the rubber is compressed in the direction of the carbon fibers, and the density of carbon fibers increases. This results in a reduced impedance under the finger which in turn results in increased current to the detecting pads under the finger. Only the detecting pads under the part of the rubber being compressed see any change in current; since the rubber has high impedance horizontally, the effect of the finger pressure is localized. This localization effect allows polyphonic detection. CONDUCTIVE SHEET RUBBER CONTAINING VERTICAL CARBON FIBERS PC BOARD WITH ARRAY OF DETECTING PADS Figure 3 The rubber in this Continuum playing surface has low impedance in the vertical direction, and high impedance in the horizontal direction. When the rubber is compressed by finger pressure, the vertical impedance is locally reduced. An increased current is detected in the detecting pads underneath the finger. Initially we had difficulty making good electronic connection at the interfaces between the rubber and the conductive sheet, and the rubber and the PC board. This problem was solved using a highquality conductive glue. The other problem with this approach we could not solve. The carbon fibers in the rubber do not last very long; after several hours of playing, the fibers break down and the Continuum stops working because the rubber becomes equally conductive in all directions. As an alternative we also tried using non-conductive rubber with Kynar (Mylar) sensors mounted on the PC board. This approach failed because of the highly sensitive circuitry required, because of difficulties in cutting the Kynar into an array of small squares, and because of Kynar's sensitivity to the heat of the performer's hands. 2.4 Hall-effect Methods Our more recent Continuum designs incorporate a simplification: they detect the independent x, y, and z (pressure) for each finger, but only one finger is detected at a given x (pitch) location. Fingers must be at least 0.3 inches apart in the x direction (a little under a half step) to be detected. This restriction still allows arbitrary and independent detection of pitch, loudness, and timbre for each note in the chord; it only disallows unisons (two notes played simultaneously with the same pitch but different timbres). The design in Figure 4 incorporates this simplification. The Hall-effect sensors measure the proximity of the magnets; when the performer applies finger pressure, the rods under the finger are depressed, and the magnets on those rods move closer to the sensors. (Before we prototyped this design we had a similar design using optical switches; Hall-effect sensors turned out to be easier to use.) Real-time software finds each finger's z location (pressure) by summing the values from the sensors of the rods under the finger. The y location (timbre) is found by the ratio of the detector values on either end of the rods. The x location (pitch) is found very accurately by cubic peak-finding interpolation between the pressures on the rods under the finger. Figure 5 shows our most recent improvement to the design. The new mounting geometry for the magnets solves a problem with Figure 4's design: the magnetic field detected by the Hall-effect sensors are now unaffected by the slight rotation of" the rods which occurs during finger movements in the x direction (glissandi), since the magnets are on the axis of rotation. 83

Page  84 ï~~1/8 IN. X 1/8 IN. RODS MYLAR--N OS SHEET 7 --,._.. MAGETSRUBBER F 0-1___HALL EFFECT SENSORS Figure 4 This design uses only two rows of sensors, one row along each side of the playing surface. This greatly simplifies the construction of the Continuum and reduces the computation power necessary for interpreting data from the Continuum. This design cannot detect unisons; all the pitches of notes in a chord must be at least a half step apart. (The fingers arc actually larger compared to the width of the rods than shown.) " MYLAR " 1/8 IN. X 1/8 IN. RODS - SHEET,.i.T MA GNETSSPRING & POST = 1 HALL EFFECT SENSORS Figure 5 The current design of the Continuum is similar to Figure 4, except that springs are used instead of rubber, and a new arrangement of magnets and sensors is used. 3. HARDWARE INTERFACE The Continuum is interfaced to a Kyma/Capybara workstation (Scaletti, Hebel, Belet 1991). The Capybara contains nine 56001 processors; this is enough processing power to (1) run the algorithm which continually computes the x, y, and z values for each finger from the sensor values, and (2) run the real-time sound synthesis algorithms controlled by these x, y, and z values. We find that only a subset of the continuous x, y, and z information for each finger can be transmitted over MIDI; MIDI data rates are not sufficient to transmit all the information in real time. In our configuration we avoid this problem by using parallel connections instead of MIDI connections. 4. CONCLUSION Any sound synthesis algorithm which makes use of the x, y, and z information can be used with the Continuum. Our interpolative Fourier synthesis algorithm (Haken 1992) is ideally suited because the x, y, and z locations can be used to interpolate between three dimensions of time-varying Fourier spectra. The Fourier synthesis algorithm produces the spectral changes which occur as the performer changes any combinatLion or pitch, loudness, and playing styles during a note. 5. REFERENCES L. Haken, "Computational Methods f'or Reall-time Fourier Synthesis," IEEE Transactions on Signal Processing, Vol 40, Number 9, October 1992. E. Johnstone, "The Rolky: A Poly-Touch Controller for Electronic Music," Proc. 1985 international Computer Music Conf'erence, pp 291-295, 1985. R. Moog, "A Mulliply Touch-Sensitive Clalvier for Computer Music," Proc. 1982 International Computer Music Conference, 1982. C. Scaletti, K. Hebel, B. Bclct, "Session 32: The Kymal/Capybara Workstation," Proc. 1991 International Computer Mutsic Contference, pp 505-516, 1991. 84