Page  599 ï~~A visual-to-sound interactive computer performance system 'Edge' Shu Matsuda Takayuki Rai Sonology Department, Kunitachi College of Music 5-5-1, Kashiwa-cho, Tachikawa-shi, Tokyo, 190 Japan jp Abstract: This paper presents an interactive computer performance system that enables visual data to control audio events in real-time. 1. Introduction Recently, the use of various different devices and computer systems that can transform the movement of the human body into data to control sound events have become an important topic in the computer music world. In the system 'Edge', movement of the human body (or any moving visual objects) can control audio events in real-time without attaching any sensor devices to the human body. It analyses bit map data of incoming images and uses the result to control sound events. By accessing bit map data in the memory of the dimension board directly, 'Edge' realizes faster interaction between visual input and sound output. It also enables visual image to control sound events precisely by the user specifying scanning points in the bit map data of the image. 2. Hardware requirements 'Edge' is realized only with commercially available equipment. No custom-made devices have been developed for or are used in this system. It consists of a CCD camera, a NeXT dimension computer(68040/25MHz), an IRCAM Signal Processing Workstation(ISPW) and a MIDI synthesizer(figure-l). 3. Outline of how it works The movement of the human body or any object in front of a video or CCD camera is sent to the dimension board which can capture the video image. The host computer's CPU keeps reading data in user-specified positions of these incoming images in the the dimension board's memory. The calculations running on the host CPU transform these data to the control data for audio events. Finally the results of the calculations are sent to the ISPW to control sound synthesis, signal processing or any musical/sound events in various MAX patches running on the ISPW. It is also possible to send data directly from the host CPU to MIDI equipment via the RS-423 serial port of the NeXT computer. 4. Scanning process The user can specify any scanning points, straight or curve lines, or any shapes of surfaces on the incoming images, that are stored in the memory of dimension board as 24 bit RGB data. User-specified scanning points are stored in main program running on the host CPU as the address of the dimension board's memory. The speed of memory access is critical in this process. Video images are refreshed every 1/30 of a second. Objective-C supplies the NXLiveVideoView class in order to enable the host CPU to access the dimension board's memory. But, since this class is written as a multi-purpose object, its accessing of the video images is too slow to realize interactive control effectively. Thus, the special object which can access the memory of the dimension board directly was developed. With this custom-made object, the host CPU can read about 15 lines in 1/30 of a second. Display for performer and audience figure-I: sustem diagram NeXT dimension computer with ISPW I C M C PR OC EE DI N G S 1995 599

Page  600 ï~~5. Scanning objects Scanning objects can be straight lines, curved lines, or point or any shapes of surfaces, and it can even be moved in real-time while capturing incoming images(figure-2). In order to show precise scanning objects for the performer as well as the audience, color scanning object images are added in the original image. The performer can perform while watching this output image and checking scanning objects. 6. Host CPU deals with scanned data The host CPU fetches the data of scanned points and after processing, sends the results to the ISPW board. The host CPU can't exchange data with the ISPW by using FTS while accessing to the dimension board directly on the same task. In order to solve this problem, the custom-made agent, so called public FTS server, which runs as another task on the host CPU, was created. By using this agent, the host CPU can realize communication with both the dimension board and the ISPW simultaniously (figure-3). 7. Controlling sound events Here is an example of image controlling sound events: When a certain point on the scanning object becomes brighter, a sound event is triggered. The brightness of the point at the triggered moment can be used as a value to control sound timbre. The amplitude of sound event can be determined by the speed of a moving object. Several ways to detect the speed of a moving object can be considered. Calculating the speed from detecting the time difference between two points can't show the speed which is faster than the sampling frequency of the video capture, 1/30. Therefore, the speed is calculated from an object passing on the straight-line distance travelled over several images in this system. When the brightness was captured on a scanning line, the host CPU refers to the memory which has kept several past data on the line and determines the speed of the object, thus the amplitude of sound event. 8. Conclusion This system looks to the future of body movement detecting music, most effective for dance performance. Because the dancer would be completely free from the extra devices for detection. Since the production of NeXT computers and the ISPW has ceased, the system must be ported to the new hardware. Right now, SGI computers can be considered as the most powerful tool for this system because of its enhanced graphic environment and MAX running on SGI. Presently, the 'Edge' is being ported to SGI. References Puckette, Miller. 1991 "FTS: A Real-Time Monitor for Multiprocessor Music Synthesis" Computer Music Journal, 15(3) 58 - 67. NEXTSTEP Developer's Library General Reference: Volume 2 6/10 Operating System Software 1/7 Addison-Wesley Publishers Japan CCD camera monitor displa video ismage::/ NeXT Dimension board read:write M.._ p -,dg-appMIDI\ figure-2: Displaying an image with scanning objects fiaure-3: 600 0ICMC PROCEEDINGS 1995