Page  127 ï~~Gesture Sensor in Virtual Performer *Tsutomu Kanamori *Haruhiro Katayose **Satosi Simura *Seiji Inokuchi *Laboratories of Image Information Science and Technology Senri LC 11f, Toyonaka, Osaka, 565 JAPAN **Osaka University of Arts Higashiyama, Konan-cho, Minamikawachi, Osaka, 585 ABSTRACT Virtual Performer is an interactive media-art environment, composed of multi-sensors, control modules which generate action from input information, computer graphic and sound generator. It interprets other musician's acoustic and motion gesture in real time, and generates music accompanied by CG. Performers and the audience can experience something new in the music, or some abstract image in the Virtual Performer's environment. This report describes multi-sensors developed in order to acquire sound and skill in playing musical instruments, especially focusing on motion sensors developed for the shakuhachi, a traditional Japanese bamboo flute. The shape of the shakuhachi is simple and has only five holes. The player's well-controlled physical actions, represented by head shaking and fingers moving near the holes, make its sound fully expressive. Our aim is to use such gestures as the controllers of accompaniment and sound effects of a shakuhachi solo. The sensors of Virtual Performer Environment do not restrict the performer's action. The output signals of the sensors are used complementarily in order to acquire data with high resolution in broad space. 1 SYSTEM Main Sensors especially for the shakuhachi System and player are the following: sensors base on supersonic measurement, gyroscope, and moisture indicator. Supersonic transmitters are equipped on a performer's hands or body. Receivers are arranged in a cube of several meters around the performer. This sensor can detect the performer's action. Three small gyroscopes are developed in order to acquire precise movement of the performer's head. Contact sensors surrounding each hole of the shakuhachi can indicate the finger's position. [1],[2] 1.1 Supersonic sensor The sensor is constructed with 8 pairs of transmitters, receivers and interval timers with I/O interface. Crimson Virtual Performer Fig.1 Block Diagram of Virtual Performer ICMC Proceedings 1993 127 1A.2

Page  128 ï~~Supersonic transmitters are put on the player. The supersonic frequency is 40KHz, and transmits channels individually. They transmit, in order of channel, supersonic waves every short duration, about fifty waves per 30ms sequentially. Supersonic receivers detect waves at the same time. A personal computer controls interval timers through I/ O port and is linked to the supersonic transmitters by wireless. The radio frequency is 300MHz. The computer sends a'start pulse' to each interval timer. It sends commands of 'channel select' and 'start signal' to a supersonic transmitter. Signals from the supersonic receivers are amplified, and converted to stop pulses for interval timers. The computer receives values from interval timers and calculates exact locations by relations of trigonometry. This sensor can detect in a 3 meter cube with 9 millimeter accuracy. (See Fig.1) 1.2 Gyroscope The second part contains *Vibratory Gyroscopes utilizing Piezoelectoricaly driving Metal Bar which is extremely downsized, highly sensitive, and easy to operate. They themselves do not have any mechanism. They have about twice as much sensitivity as mechanical gyroscope which has been generally used. Three transducers attached to the head-band of the performer continuously output three-axis angle velocities at 50Hz Cycle. Signals,0-5 Volts, are converted to 8 bit serial binary data by an AD converter and a 16 bit single chip processor. The data is transmitted to a consolidation module by wireless. The radio frequency is 150MHz. (See Fig.2) (* Murata Seisakusho produces this sensor.) 1.3 Moisture indicators This sensor is generally used detect the moisture in wood, paper etc. however we used it to detect a resistance of the human's body. We have used this sensor to detect the contact of the player's finger at present. We are now engaging in producing these sensors to obtain gradual control of fingers. When these sensors are attached to a hole of the shakuhachi, it can detect a finger's pressure around the hole linearly. However a condition of the Human's skin continually changes by his physical condition or humidity. We have been constructing sensors based on the idea of dynamic control of thresholds. 15OMHz FM Fig.3 Block Diagram of Console V MIDI Out Yn 24Vp-p Ii_ _.L _ 30ms.i Fich 1 ch Fig.2 Block Diagram of Supersonic Sensor 1A.2 128 ICMC Proceedings 1993

Page  129 ï~~150MHz FM Gyroscope Fig.4 Block Diagrum of Gyroscope Sensor Every hole of the shakuhachi has four metal contact sensors. They are connected to high-gain amplifiers. When the performer touches the sensors, they can indicate a very small amount of leak between him and the ground. The system can obtain information of the finger's positions by four sensors. The data is transmitted by wireless (150MHz). (See Fig.3) 1.4 Other sensor In addition to these three sensors, an image processing sensor, which detects the light of the light-emitting diodes attached to the performer's body is used in order to acquire rough movements. The infrared filter attached to a camera enables us to use this sensor in the ordinary lighting environment. When the lens is changed, it is possible to alter its range. 2 CONCLUSION This paper has described sensors of Virtual Performers. In the system we combine the sensors which are supersonic sensor, gyroscope, and moisture indicator. They realize the high resolution measurement in a short distance. The images processing sensor take change of the rough resolution. It is difficult for machines to recognize outside information as humans do. Complementary utilizations of multi-sensors make it possible to realize human-like perception in the Virtual Performer. The sensors do not detect the breath of the performer at present, however our system has an acoustic sensor. It can detect most of the breath control in a playing the shakuhachi. We are planning to develop a Synthesized Shakuhachi with a breath sensor. 3 Acknowledgment The authors of this paper thank to Mr. Shikatsugu Oyagi for making the touch sensors. References [1] SIMURA,Satosi. "Oto no ibuki: syakuhati soho no seisei to henka [The sound of breath:development and change in syakuhati techniques]". In FUJII, Tomoaki; YAMAGUTI, Osamu; TUKITANI, Tuneko(editor), Gaku no utsuwa [Instruments for music making], (Tokyo:Kobundo):pp.138-148,1988 [2] TOYA, Deiko. "Komuso syakuhati sinan[Primer for the syakuhati of komuso, medicant priest of emptiness]".Hukuoka(Fukuoka): 1984 150MHz FM Amplifier Fig.5 Block Diagram of Touch Sensor ICMC Proceedings 1993 129 1A.2