Page  00000032 "It's SHO time" -- An Interactive Environment for SHO(Sheng) Performance Yoichi Nagashima Art & Science Laboratory ( 10-12-301, Sumiyoshi-5, Hamamatsu, Shizuoka, JAPAN Tamami Tono Ito Keio University SFC ( Abstract: This paper is intended as an investigation of new interactive environment for SHO (Japanese traditional instrument, a mouth organ like "Sheng" in China) performance. We have developed three styles of sensors for SHO performance with bio-sensor technology and microelectronics, designed a pattern recognizing module, and produced a multimedia / interactive environment for composition / performance of computer music featuring SHO. We also experimentally composed and performed some new works as multimedia arts, so we will report the performances and discuss about its problem and its potential. 1. Introduction The SHO (Japanese traditional instrument, a mouth organ like "Sheng" in China) player blows into a hole in the mouthpiece, which sends the air through bamboo tubes which are similar in design and produce a timbre similar to the pipes in a western organ. It can produce chords as well as single notes. It is important and interesting that the SHO player uses both directions of the breath stream, and controls the breath pressure for expressions in music. We have developed three styles of sensors for SHO performance with bio-sensor technology and microelectronics: (I)small control switches in the SHO, (2)bidirectional breath sensor, (3)wireless module for these sensors, and designed a pattern recognizing module in MAX, and produced a multimedia / interactive environment for composition / performance of computer music featuring SHO. The SHO player uses both hands to hold the SHO and performs the "fingering" to generate many notes with many fingers. Normally the player must keep sitting calmly, so we cannot use popular interfaces like foot pedals, foot volumes and optical beam sensing the movements of arms. 2. The 1st SHO sensor module At first, we found that only one finger or two fingers can be used for other purpose in the performance. We developed very small touch switches inside the SHO bamboo tubes secretly, and the player can control them within the traditional way of fingering in music. The control information generates MIDI signals with original format by small CPU module, and real-time processed with original MAX patch of pattern recognition. T^.P^br# __ The 1st SHO sensor The circuit of the 1st SHO sensor - 32 - ICMC Proceedings 1999

Page  00000033 This CPU card,AKI-80J has 8-bit integrated CPU core, and is very easy to produce special sensors. The documents of this CPU card and MIDI applications are opened at the Web: [only in Japanese now]. 3. The 2nd SHO sensor module Next, we developed a compact breath sensing system of the SHO. The breath stream of each bamboo tube is very critical, and it is very difficult to detect the value of the bi-directional pressures for each bamboo pipe. SWe found that normal SHO uses 15 bamboos with reed but 2 bamboos are used only for decoration, not used for sound generation. So we replaced one bamboo with the "sensing pipe" which connected a small air pressure sensor module. This sensor detects the bi-directional air pressure value of the "air room" of the bottom of the SHO. We also produced an original MAX patch to detect and convert the sensor information into universal parameters of performance. the pressure sensor block of the 2nd SHO sensor AKI-H8 Xtal = 16.00MHz 74HC05 CNI 1 ON04 220ohm ------- E --------------- M --- 25pinm ME +5V EM I 220ohm - E --- +----E --- 2 2 3 4 Co 5 S ^~i ^> +--------- 1 @ @3 MAX232(0) MIDI OUT 5pir DIN 11.12pinO / 10Kohm (MJDI D7@ ) CN2 12pin ----------------- -A/ in 1 smart 32bit CPU card [ AKI-H8 ] the circuit of the 2nd SHO sensor BAtO3Z(+ imi too IJfl - J, F4 2b the development system the circuit of air pressure sensor ICMC Proceedings 1999 -33 -

Page  00000034 This CPU card [AKI-H81 has 32-bit integrated CPU core and 8-channel A/D converters, and is very easy to produce advanced special sensors. The documents of this CPU card and MIDI applications are also opened at the Web [only in Japanese now]. [ metro, 5o, adb1ng:4. The 3rd SHO sensor module touchi We have developed the 3rd SHO sensor that contains both small control switches and breath sensing block, and this sensor does not 80. have "wired" demerit. We use the wireless transmitter / receiver, I so the performer can move anywhere with the wireless and the battery. so 5. Detecting SHO information with MAX The output of these sensors are mapped to special messages of MIDI, so we can easily use the messages with MAX patches as the composition. This figure is a sample patch to display the breath. sensor in real time. The left windows shows the straight breath data of the sensor [0-127], and the right window shows the. - abstract value of the bi-directional information. The sampling rate of the breath sensing is about 200Hz. The CPU can detect at 125KHz at maximum, but the MIDI receiver cannot receive the speed, so the CPU software detects the change of data. MAX patch for breath data 6. The performance We will report two applications of the interactive environment for SHO, as some works of computer music and multimedia art. 6-1. "dinergy 2" One of the authors (Tamami Tono Ito) composed and performed a work called "dinergy 2" in the concert of ICMC 1998. This piece uses the first sensor, and the performer controls "scene changes of back-grounded part" and "patch changes of Kyma" for signal processing. Please read again the documents of the concert program of ICMC1998. 6-2. "Visional Legend" One of the authors (Yoichi Nagashima) SHO sensor SHO Microphone composed and Tamami Tono Ito performed a Swork called "Visional Legend" in the concert of International Computer Music Festival 1998 Kymaa R in Kobe Japan (Xebec Hall). This piece uses ym Capy ara L the second sensor, and the performer controls (/F Ky ma DSP) ---- -:_ - S not only sound generation but also backM lol Mo grounded graphics (CG, Video, CCD image) winrYo P PA in real-time. This composition is inspired withthe poem "E1 r,K3tf: ~Ai'" written by R Shimpei Kusano and the SHO sounds MAX CD Player L __. performed by Tamami Tono Ito. The music F J- part of this piece contains two types. The SSHO sound is directly amplified to PA and is Prok -_ real-time processed by Kyma signal processing workstation with live control via "Black" NTSC snagerrator MIDI Video Projec MID. The SHO performer may control the 3CCDcameras for liie frformance ----- Video -.:::: "special breath sensor for SHO" produced by 3 Hi-8 players for bac;jroung ideo------ Switcher NlSC:-----; the composer, and the sensing information also LUe comput9rgraphcs(Indy/PC)----- modify the live SHO sound part. The I__background sound part is pre-processed with Kyma and Indy, and fixed to CD. All sound -34 - ICMC Proceedings 1999

Page  00000035 materials of this part are SHO sound performed by Tamami Tono and reading speech of the poem by Junya Sasaki (Baritone). Both sounds are processed by different algorythm in Kyma. These processed materials are processed in Indy by original DSP softwares produced by the composer, and re-mixed in the Indy. The graphical part of this piece is considered as the environmental images of the poem. There are some sources of the graphical part, for example, live slide show images of Japanese scenary, CG generated by Open-GL softwares, natural video images and live graphics of the performer with CCD cameras. These source images are live switched by MIDI-Video switcher, and the output signal is connected to a projector for big screen. The SHO performer may switch the graphics with the special SHO sensor. The algorithm of real-time composition and graphical control is realized as a MAX patch. The SHO performance is described in this score, but the most important policy is the improvisation. The performer may have a Stop Watch to detect the position of the piece with this score, but playing points are not fixed exactly. She/He may perform with the improvisation spirit of her/his own, and she/he must listen to the sound in the whole performance.,t,,,oe.1 60 6w~lo 1 4 Ito I2,0=40~ 4 " Kyma Patch for "Visional Legend" The performence of "Visional Legend" 7. Summary We have reported some examples of our research in this paper. The sensor technology may apply not only for music but also human interface systems. We will continue this research both with another sensors for music and with human interface applications. REFERENCES [Nagashima95a] Nagashima, Y. Multimedia Interactive Art: System Design and Artistic Concept of Real Time Performance with Computer Graphics and Computer Music. Proceedings of Sixth International Conference on Human Computer Interactio.Elsevier, Yokohama Japan, 1995. [Nagashima95bl Nagashima, Y., Katayose, H., Inokuchi, S. A Compositional Environment with Interaction and Intersection between Musical Model and Graphical Model --- "Listen to the Graphics, Watch the Music" -. Proceedings of / 995 International Computer Music Conference. International Computer Music Association, Banff Canada. [Nagashima98a] Nagashima, Y. Real-time Interactive Performance with Computer Graphics and Computer Music. Proceedings of IFAC Man-Machine Systems International Conference. Elsevier, Kyoto Japan, 1998. [Nagashima98bl Nagashima, Y. BioSensorFusion: New Interfaces for Interactive Multimedia Art. Proceedings of I~aasimaSa Naasima Y Mutiedi iterctve rt: Sstm Dsin ad rtiti Cocet o r4)yim Perfrmace wth ompuer raphcs nd Cmpuer Msic Proeedngs f Sxth ntenatinalin &,noe o H~mn apHam am Apo.lsver okhmaJpa,195 I~a~shia95b Naashma,., ataoseH.,Inokchi S.A Copostioal Evirnmen wih Iterwas aan 1998 International Computer Music Conference. International Computer Music Association, Michigan USA. ICMIC Proceedings 1999 - 35 -