Page  265 ï~~Sensor Integration for Interactive Digital Art Tsutomu Kanamori, Haruhiro Katayose, Yushi Aono, Seiji Inokuchi and Takasi Sakaguchi Laboratories of Image Information Science and Technology Tel: +81-6-873-2030 /E-mail: ABSTRACT: In this paper our recent activities concerning sensor integration in interactive digital art is described. The approach we employed for the information input from performers of interactive art is the multi-modal utilization of sensors. This paper introduces a posture sensor based on acceleration transducers and an image sensor designed to detect remote positional data. This paper also introduces the console unit for effective utilization of multi-modal sensors. 1. Introduction Interactive performance is one of the most significant themes in the field of computer music (Katayose,H.)(Kanamori,T.)(Tanaka, A.). We have developed a total composing environment for this interactive art, especially elaborated through sensor development. Our approach to sensor development is to give sensors multi-modal functions. Multi-modal sensors are very useful in a variety of arts. However, some problems arise and we should prepare countermeasures; data traffic control, standardization of data format, reduction of performer's load. In this paper, we discuss the standardization of sensors from the viewpoint of hardware and some new developments of the sensor unit. 2. Sensor System Overview The sensor system is divided into the transducer parts and the console unit. The transducer parts include two types. Table 1 shows the new developed sensors. (See (Kanamori,T.) for the other sensors.) Table 1 Transducers feature Posture Sensor Range Output Freq. Range Defect Gyroscope AC 0.1--20Hz DC drift * Acceleration Sensor Narrow Range Type -1.5-+1.5G DC 0-10Hz Direction** Wide Range Type -20-+20G AC 1-4000Hz Direction** *DC drift at a static posture. **Detection incapability of dynamic direction and rotation Image Sensor Infrared CCD camera NTSC 0-6MHz View area*** *** Depends on optical system. Our desire is that many kind of sensors should work complimentarily. Gyroscopes are good at detecting rotational movement, but bad at detecting static angle. Acceleration transducers make up for this weak point of the gyroscope. It is desired to acquire the position of the plural moving parts of a performer or ICMC PROCEEDI NGS 1995265 265

Page  266 ï~~performers during a more sophisticated performance. We have been developing an image sensor for this requirement. To standardize the data format of various sensors, we have designed a console unit. In the following chapters, we illustrate a posture sensor, an image sensor and a console unit. 2.1 Posture Sensor Acceleration transducers are used in automobiles and industrial robots. They are designed for detection of impact or static posture. The acceleration transducers which we employed for the development of the sensors are the following two kinds, narrow range type and wide range type (Mitsubishi electric co.). The former type is designed to detect gravity and can be used to measure posture angle below 10 Hz movement. By arranging these transducers along two or three points along the axis of the performer's body, we can consequently get the posture data of the position where the transducers are attached. The outlook of the posture sensor and the principle of the measurement are shown in fig. 1 and fig. 2 respectively. One defect of this sensor is its incapability to detect horizontal direction. The sensor based on the gyroscope is used to make up for this defect. The second transducer is designed to detect impact. We use this transducer to detect a performers' gesture such as jumping. This transducer is also used for the calibration of the sensor based on the gyroscope. The output of the gyroscope is the angular velocity which contains a trend wave of some low frequencies. Zero calibration at the static status is required. The acceleration transducer is used to judge if the gyroscope is in static status or dynamic status. Acceleration Transducer 2Silicon Strain Gauge Lowpass Filter -12dB/Oct. Silicon Oil Weight X-axis Block Y-axis Block _ _ _ 5 ------Base Configuration Data External Pin Standard plug To l6bit Single Chip Processor Fig.l. Diagram for Posture Sensor Fig.2. Mechanisms of Acceleration Transducer 2.2 Image Sensor The advantage of image sensors is the remote detection of positional data by finding the concerned pixel in an image Special matching processing is not required for the detection of the one concerned position 266 I C M C P ROCEE DINGS 1995

Page  267 ï~~represented by the brightest pixel. However, for the detection of plural positions, identification problems arise. To cope with this problem, we have adopted an active light control method. Our image sensor consists of an infrared CCD camera, plural infrared light-emitting diodes which are attached on the performer's bodies and a 16bit single chip computer. The sensor can identify plural diodes, because the system knows the timing of light-emittence which is actively controlled from the video signal directly. Ordinary image sensors require a frame memory to handle pixels in an image. It results in high cost and non-portability of image sensors. We have achieved pixel identification directly from the video signal without using frame memory, by counting the time from the H-sync/V-sync signal to the instance when concerned position is detected by the binarization processing. The infrared bandpass filter attached to the camera and the floating binarization for the detection of the light-emitting point enables the sensor to work in an variable light environment. One of the other characteristics of the sensor is the interpolation method for the pixel identification. It contributes to the improvement of positional resolution. If the system uses two or more cameras, it will be able to detect 3-Dimension information. iiiiiiiiiiil i -sync,,Reference Voltage..........................................................,.................:::::::::::::::::.................................. Comper F ii iil ig.. Ifae C amr n rcso FInt4uDiagrames Seofnmg Sno:::::::::::::::.................................................;:..:;;;:;;;;;:;;;;:;;::::;;;::;;::;::::;;;: z::;;;:::;:;::;:;:..........;;;;::......... +.............;::: +: +................................. Fi.??.?:? SycgS4. Detectrmo iaeSno 3. Console Unit We use the various sensors, necessary to absorb complexity and to provide a good environment for creativity. It is desired that data from sensors be uniform. In the console unit, the output of sensors are transformed into sequential digital values and transmitted to the other systems using a wireless system. A l6bit single chip computer controls the console unit in a small box with a Ni-Ca battery attached to the performer. The console unit and the transducer parts provide an adjustable facility of output data for various sensing requirements. The users can easily adjust the gain, offset, resolution and sampling rate. I C MC PROCEEDING S 199526 267

Page  268 ï~~The hardware configuration of sensors is also controllable. The connectors of each transducer to the integration module are designed to have the standard plug and the user can easily set the appropriate sensor configuration. The console unit is a fault tolerant system. The plugs are allowed to be connected or disconnected at any time without a power shutdown, because ground lines and power lines are connected sequentially and all lines have protections against short circuit and static electricity. Each plug has input lines for 3 analog signals, 3 digital serial signals and attribute lines. The 8 attribute lines form 8bit binary data. The console unit dynamically selects the configuration of the sensors. The analog signal line is surrounded by the ground lines. This line can receive 0 to 5volts. The users can set up the console unit at any baud-rate, any resolution and any sampling rate. Vcc Vcc GND Standard plug Analog 1 GND Digital Data Analog Data GM2 GND Analog 3 Serial Port AD-Convertor GND Digital 1 Clock 1 Digital 2 16bit Single Chip Computer Clock 2 Digital 3 Clock 3 Attribute 2 Attribute 1 Attribute 4 Attribute 3 TransmitterAttribute 6 Attribute 5 Attribute 8 Attribute 7 Reserve Reserve Vcc Vcc Fig.S. Console Unit Structure Fig.6. Standard plug for the Console Unit 4. Summery This paper introduced a posture sensor based on acceleration transducers and an image sensor designed to detect remote positional data. This paper also introduced the console unit for effective utilization of multi-modal sensors. They are expanding the artistic freedom of interactive art. We are now conducting a project to utilize the developed sensors for interactive art, especially focusing on multimedia art featuring dance. We would like to develop further systems and also to produce various kinds of electric art pieces. References (Katayose, H) Katayose, H. Kanamori, T. Kamei, K. Nagashima, Y. Sato, K. Inokuchi, S. and Simura, S.:Virtual Performer, Proc. ICMC'93, pp.138-145, 1993. (Kanamori, T) Kanamori,T. Katayose, H. Simura, S. and Lnokuchi,S.:Gesture Sensor in Virtual Performer, Proc. ICMC'93, pp.127-129, 1993. (Tanaka, A.) Tanaka, A." Musical Technical Issues in Interactive Instrument Technology with Application to the BioMuse, Proc. ICMC'93, pp.124-126, 1993. 268 8ICMC PROCEEDINGS 1995