Page  00000001 Software Sensors for Interactive Digital Art Shigeyuki HiraLi Haruhiro Katayose Tsutomu Kanamori and Seiji Inokuchi L.I S T. DaiichiKasaLi-Senrichuo Bldg., ShinsenriNishi-machi, Toyonaka, Osaka, 565-0083, JAPAN hirai @ Abstract Recent PCs are getting more powerful and having multimedia functionalities for Audio/Video inputs. We propose Software Sensor, utilizing the AV inputs as A/D devices and higher level sensors with Audio/Video DSP. System designers and artists of interactive digital arts (computer music performances, multimedia installations, etc.) will be able to develop interactive systems which have more various sensors utilizing the Software Sensors. This means interactive system, which does not use complicated hardware sensor systems except transducers, will be simpler and easier to develop and manage. This paper describes some of the Software Sensors implemented as MAX external objects on Macintosh platform and some applications with them. 1. Introduction Recently, a number of multimedia contents are produced. Interactive Digital Art is no exception. Many people who concerned with them have developed various hardwares as input devices, including transducers and equipments, for instance Spirit Chair [Paradiso 1997], Digital Baton [Marrin 1997], GraspMIDI [Sawada 1997], etc. Useful information on other input devices is given in "The Computer Music Tutorial" [Roads 1996]. On the other hand, some sensor systems are designed for general-purpose real-time performance systems, for instance I-Cube System [Mulder 1995], ATOM8 [Katayose 1996], Fish [PaLradiso 1997] and Leider's system [Leider 1997]. As these systems have microcontrollers and A/D devices, system designers can devote to think transducers for handling performances. These hardware sensors have contributed to the dispersion of load charged on the host computer. But many systems are not ease-to-manage, because of many hardwares, transducers and sensor systems units. Nowadays, we can get and use powerful PCs equipped with Audio/Video peripheral devices. If the devices are utilized as A/D devices for the various transducers, we can make simpler and more compact systems without hardware sensor units. It's not problem to perform the A/D tasks for the latest PCs and to control sampling rate and size with software. In addition, these PCs and software can also perform as higher level sensors with DSP. Here, we propose these tasks as 'Software Sensors' and describe as follows. 2. Software Sensors 2.1 Utilizing audio inputs as A/D devices As mentioned before, the latest PCs have Audio/Video input devices generally. Assuming that the input devices are A/~D devices, we can say a microphone is one of transducers for audio input, and a CCD camera is one of transducers for video input. If several subjects are settled, an audio input can be a general input for analog signals from various transducers. One of the subjects is to amplify the signals to line-level signals. Another subject is to assume that the audio input does not accept static signals (DC signals) because of a highpass filter as DC canceler. These subjects are not serious, if we utilize a DC-AC circuit like a Ring-Modulator (see figure. 1) and a demodulator (AC-DC) software. This utilization for controlling A/D is one of the origin of the name 'Software Sensor'. Also, on Macintosh platform, most of PowerMac have a sound input with an electric source connecting to a PlainTalk microphone. Some transducers can drive with this source without pre-amp. This means that utilizing the Software Sensors makes sensor systems very simple, easy-to-develop, and even changing paradigm of designing interactive systems.

Page  00000002 input 4027B ~ output 4 8 Q Q 555 Q__ Figure. 1 DC-AC modulator 2.2 Software Sensor as MAX external object About software environment, on the other hand, the most typical approach adopted in the projects of interactive digital arts is utilization of visual programing environment represented by MAX. The disparity between MAX and other development tools is to describe data flow of real-time continuous valued data on MAX. This point is very simple and smart to develop an interactive system. If the A/D process and various high level sensors with DSP are available on MAX environment, many people who develop interactive systems will be able to develop easily, and be saved the trouble of hardware maintenance. As a result, the Software Sensors have been implemented as MAX external objects on the Macintosh platform for the time being. There are recently MSP and Pd/GEM [Puckette 1997] [Danks 1997], these software environments are touched on later. We introduce two Software Sensors with DSP as for example in this section. 2.2.1 Pitchsensor --------------- The Pitchsensor object using the Macintosh Sound Input Man- pitchsensor Object Thisobect etectthepitchandpower copy right ~ 1997 H.Katay ose, S.Hirai our singing voice through Macintosh ager (at 22kHz, 8bit sampling) is one of acoustic sensors with Check his to detect yourSampling Rate 22kHz SpSampling Size 8 bits audio DSP. This object detects pitch and power from acoustical 00 Note This obect require powechine Pitch Data Power Data signals of singing voices or lead notes of various instruments via 400 Hz a microphone. There are two versions of this object, both are compiled as FAT I I objects. The first is based on integral zero-crossed method that 100HI 1 jluII IjjIi0 calculate the period of zero-crosses from integrate sampled data. 255 The second is based on self-correlation of sampled data. We implimented these for the time being, however, there are many studies concerned with pitch detection. It is easy to get and display the pitch and power data with this Figure.2 pitchsensor object (MAX help patch) Software Sensor as figure.2. 2.2.2 DigitEyeClassic The DigitEyeClassic is a 3D Motion Capture apparatus. The functionality of this external object has originated from our 3D motion capture technology DigitEye3D. The technical issues of the original system are as following section, and DigitEyeClassic as a Software Sensor is mentioned then. DigitEye3D The DigitEye3D is a simple, optical type, 3D motion capture system Figure.3 DigitEye3D

Page  00000003 consists of an original hardware box, several spherical markers with infrared LED, a monochrome CCD camera has a filter passes infrared rays. This system can distinguish each marker by controlling the flash with NTSC vsync signals of a CCD camera. Each position is calculated from light area of a field without a frame-buffer. 3D positional data are sent to a PC via a serial port (RS-232C). In addition, this simple system is fairly inexpensive comparing with many commercial 3D motion capture systems. See figure.3. DigitEyeClassic as MAX external object @n itgitEyel~ssichlp e The DigitEyeClassic is a restricted DigitEye3D system DigitEyeClassic Object Attention: ver 0.32 Outputs is NOT a accurate 3D position. as a MAX external object. This handles a video flame cop right O 1998 S.Hiral ot accurac, itshould table. from a CCD camera via Macintosh video input with DigitEyeClassic requires - a powerful Macintosh with Video Input QuickTime Components. The original DigitEye3D hard- a CCD.amra with IR pass Filter S a penlight ware box is unnecessary, though this object can identify DigitEyeClassic grabs a video frame and calculates only one marker. The Macintosh which DigitEyeClassic the position of a flash area. runs should be powerful because of the load of video DSP. As a result, it is easy to develop and manage interactive Horizontal Vertical Diameter systems with a 3D motion capture sensor. See figure.4. Figure.4 DigitEyeClassic object (MAX help patch) 3. Applications Using Software Sensors In this chapter, we introduce applications using the above-mentioned Software Sensors as MAX externals. 3.1 VSG (Voice Shooting Game) This application utilizing the Pitchsensor object is a 3,EF d Voice Shooting Game MAX application, we have been developing Singing. Training System for poor pitch singers. This system is High Score an interactive system based on a real-time visual feedback of singing pitch and power. Several modes, game [ mode, trace mode, karaoke mode, etc. are available. There was an original hardware unit we developed to T AA detect vocal pitch. It was also difficult to use for ordi- t~ithE nary people. The VSG system at present becomes ease- I 2 to-use, because of embedding the Pitchsensor object in................ the VSG MAX patch. Figure.5 VSG 3.2 Play the D.E. Play the D.E. is a sample application as a new instru- lptinn-s a sample application as a new instru- F E Fe tt-id T1P1 ment with DigitEyeClassic. The 3D position of a marker PLAY BGM 1 0 makes notes. The horizontal position corresponds to a PLAY DigitEye / pitch (MIDI note number), the vertical position corre- s sponds to volume (MIDI velocity), and the distance E E from CCD camera related to density of notes. In addi-................................... tion, the notes can be played along several scales, Bluenote-Scale, Ionian-Scale, Pentatonic-Scale, etc. 4. Future Developments As the Software Sensors we implemented are only a:....: ~~l~~~~~~I~~~~~l~~~'~I ~ II~ll''' t~---.l---------- ~ i few sensors, we will develop more various sensors with DSP like formant sensor, vowel sensor, and so on. And

Page  00000004 now, there are MSP on Macintosh which can handle multi-channel audio inputs as A/D devices and real-time audio DSP, and Pd/GEM on Windows95/NT, Linux and IRIX(SGI) which can handle not only audio but also video DSP. It is not necessary to write a part of programs to control sound input devices, and we can concentrate to write DSP programs on these environments. So we plan to port these sensors to MSP objects for the present, and Pd/GEM object in the future. 5. Summery This paper describes concepts of the Software Sensors and two implemented sensors with each application. The first concept is to use latest PC's AV inputs as A/D devices and the second is to implement sensors with DSP as MAX externals for ease-to-use. The Pitchsensor object with audio DSP and the DigitEyeClassic object with Video DSP are based on these concepts and available actually. The technology we proposed is very important for interactive multimedia contents, as well as interactive digital arts. We would like to contribute for system designers and artists to make easier to develop and manage systems using the Software Sensors. References [Danks 1997] M.Danks. Real-time Image and Video Processing in GEM. Proc. ICMC, pp.220-223, 1997. [Kanamori et al. 1995] T.Kanamori. et al. Sensor Integration for Interactive Digital Art. Proc. ICMC, pp.265-267, 1995. [Katayose et al. 1996] H.Katayose. et al. An Environment for Interactive Art. Proc. ICMC, pp.173-176, 1996. [Katayose et al. 1997] H.Katayose. et al. A Toolkit for Interactive Digital Art. Proc. ICMC, pp.476-478, 1997. [Leider 1997] C.N.Leider. A Realtime MIDI Library for an Inexpensive General-Purpose Microcontroller-Based Interactive Performance System. Proc. ICMC, pp.482-484, 1997 [Marrin 1997] T.Marrin. and J.Paradiso. The Digital Baton: a Versatile Performance Instrument. Proc. ICMC, pp.313-316, 1997. [Mulder 1995] A.Mulder. The I-Cube System: moving towards sensor technology for artists. people/ResearchStaff/amulder/personal/infusion/ISEA95.html, 1995. [Paradiso 1997] J.A.Paradiso. and N.Gershenfeld. Musical Applications of Electric Field Sensing. Computer Music Journal, 21:2, pp.69-89, 1997. [Puckette 1997] M.S.Puckette. Pure Data. Proc. ICMC, pp.224-227, 1997. [Roads 1996] C.Roads. The Computer Music Tutorial, The M.I.T. Press, 1996. [Sawada 1997] H.Sawada. et al. Sounds in Hands -A Sound Modifier Using Datagloves and Twiddle Interface-. Proc. ICMC, pp.309-312, 1997.