Page  00000001 Combined Force Display System of EMG Sensor for Interactive Performance Yoichi Nagashima Shizuoka University of Art and Culture / Art & Science Laboratory nagasm @computer.org Abstract This is a report of research and some experimental applications of human-computer interaction in computer music and interactive media arts. In general, many sensors are used for the interactive communication as interfaces, and the performer receives the output of the system via graphics, sounds and physical reactions of interfaces like musical instruments. I have produced many types of interfaces, not only with mechanical/electrical sensors but also with biological/physiological sensors. This paper is intended as an investigation of some special approaches: (1) 16-channel electromyogram sensing system called "MiniBioMuse-III" and its applications, (2) 8-channel electric-feedback system and its applications, (3) combination of EMG sensor and biofeedback system sharing same electrode to construct the "force display" effect of live control with EMG sensors. 1. Introduction As the research called PEGASUS project (Performing Environment of Granulation, Automata, Succession, and Unified-Synchronism), I have produced many systems of realtime performance with original sensors, and have composed and performed many experimental works at concerts and festivals. The second step of the project is aimed "multimedia interactive art" by the collaboration with CG artists, dancers and poets. Fig.1 shows the concept of the project [1-13]. Scenario (Artist Composer) SoundInput Sensor Videolnput Parameter Fusion Parameter Messages Control System (Time / Space / Structure) Messages "Sound World" Model "Visual World" Model (Sound Generate Agents) (Graphics Generate Agents) Acoustic I Pixel Level I I Level Database Database Note MusicTheory | Object VisualTheory Level I Music Psychology Level 3D Modeling Visual Psychology Composition Composition Sound Outut Visual Outut Figure 1: Conceptual system block diagram of PEGASUS project 2. "MiniBioMuse - III" At first I report the development of a compact/light 16 -channels electromyogram sensor called "MiniBioMuse-III" (Fig.2). This sensor is developed as the third generation of my research in electromyogram sensing, because there are many problems in high-gain sensing and noise reduction on stage (bad condition for bio-sensing). Figure 2: "MiniBioMuse-III" (only for one arm) 2-1. Development of "MiniBioMuse-III" The front-end sensing circuit (Fig.3) is designed with heatcombined dual-FETs, and cancels the common-mode noises. There are 9 contacts on one belt, one is common "ground". Figure 3: Front-end circuits of "MiniBioMuse-III" Figure 4: High-density assembled circuits (8ch)

Page  00000002 Each 8-channel electromyogram signals for one arm/hand (fig.4) is demultiplexed and converted to digital information by 32bits CPU, and converted to MIDI information for the system (fig.5). This system also generates 2 channel Analog voltage outputs for general purpose from MIDI inputs. )/Affit Figure 9: MIDI output of "MiniBioMuse-III" 2-3. Performance with "MiniBioMuse-III" I have composed a work with this "MiniBioMuse-III" for my Europe Tour in September 2001. The title of the work is "BioCosmicStorm-II", and the system is constructed in Max/MSP environments. 16 channel electromyogram signals are all displayed in Max/MSP screen and projected on stage, so audience can easily understand the relations between sound and performance. This work was performed in Paris (CCMIX), Kassel and Hamburg. Generated sounds are three types in scenes: (1) 8+8 channels bandpass-filters with white noises, (2) 16 individual-pitch sine-wave genarators, and (3) 3+3 operators and 10 parameters of FM synthesis generators. All sounds are real-time generated with the sensor (Fig. 10). Figure 5: 32bits CPU circuits of "MiniBioMuse-III" This CPU also works as software DSP to suppress the Ham noise of environmental AC power supply. Figure 6 shows its algorithm of "Notch Filter" for noise reduction. Figure 6: Noise reduction algorythm 2-2. Output of "MiniBioMuse-III" Figure 7/8 shows the output analog signals of front-end circuits of "MiniBioMuse-III". Figure 7 is the "relax" state of the performer, and figure 8 is "hard-tension" state of the performer. V dB 0.5- - - -..0.4 -30: -40 0.2:: - 60:::: - S60: - o0.3........... i.........;...o;....... -0.4 7........ S10 20 30 40 50 60 70 80 90 100 0.0 0.2 0.4 0.6 0. 1.0 1.2 1.4 Figure 7: Sensing output signals of "relax" V dB 0.5 0.3 []0 10 20 30 40 50 60 70 80 90 100 I 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Figure 8: Sensing output signals of "hard tension" Figure 9 is the Max/MSP screen of the MIDI output of this sensor. 8channels + 8channels electromyogram signals are all displayed in real time, and used for sound generating parameters. The maximum sampling rate of this sensor is about 5msec, but this sampling rate is changed by the host system for its ability of MIDI receiving. Figure 10. Performance of "BioCosmicStorm-II" I have composed new work with this "MiniBioMuse-III" for NIME2003. The title of the work is "Quebec Power", and the system is constructed in Max/MSP environments. This work was performed at McGill University in Montreal. All sounds and graphics were real-time generated with the sensor and live performance of Bass Recorder (Fig. 11). Figure 11. Rehearsal of "Quebec Power"

Page  00000003 3. Bio-Feedback System Secondly, I report the newest development of a compact and light 8-channel biological feedback system (Fig.12). The feedback signal is high voltage (10V-100V) electric pulses like "low frequency massage" device (Fig.13-14)). by Masayuki Sumi in February 2002 in Japan. The performer is a professional dancer, so he can receive two times bigger electronic pulses with his strong and well-trained body. The composer produces many noises and sounds with Max/MSP, and the converted signals control the body of the performer on stage. The relations of them are well shown in real-time with the screen and motions on stage. 3.3. Application Example 3 Figure 17 shows the performance of the work called "Ryusei Raihai" composed by Masahiro Miwa in March 2002 in Japan. The four performers connected to the system are "instruments" of the special message in Internet with the composer's filtering program. When one special data occurs in the network, one of the performers is triggered by the system, then he/she plays bell on the hand in real-time. This work was performed in some festivals in Japan and Mexico. Figure 12. Bio-Feedback System The waveshape, voltage and density of pulses are real-time controlled with MIDI from the system. The purposes of this feedback are: (1) detecting performer's cues from the system without being understood by audience, (2) delicate control of sounds and graphics with the feedback feeling in virtual environment, (3) live performance of outside of anticipation with the electric trigger. W.......... Figure 15. Performance of "It was going..." Figure 13. Example of Bio-Feedback signal Figure 16. Performance of "Flesh Protocol." Figure 14. Bio-IFeedback contacts 3.1. Application Example 1 Figure 15 shows the performance of the work "It was going better If I would be sadist truly." composed and performed by Ken Furudachi in February 2002 in Japan. There were 2 DJ (scratching discs) performers on stage, and the DJ sounds generates many types of bio-feedback signals with Max/MSP and this system. The performer shows the relation between input sounds and output performance just by his body itself. This work is the first application of the system. 3.2. Application Example 2 Figure 16 shows the performance of the work called "Flesh Protocol" composed by Masayuki Akamatsu and performed Figure 17. Performance of "Ryusei Raihai" 4. Combined force display system In experiments during development of the bio-feedback system, I found many interesting experiences to detect "sounds" without acoustic method (headphone, speaker, etc). The numbed ache from this bio-feedback system is different

Page  00000004 with the waveshape, frequency, density, interval and polarity of pulses. This shows the possibility that bio-feedback can be used as the "force display" of EMG performance in music as instruments. 4.1. Time-sharing of electrode It is well known that EMG sensor is a good instrument, but there is a weakpoint compared with other "mechanical" instruments. EMG sensor does not have the "physical reaction of performance" like guitar, piano, etc. To resolve this, one musician generates very big sound as the body-sonic feedback of the performance. I have just started researching the combination of EMG sensor and bio-feedback system with the same electrode using time-sharing technique. Figure 18 shows the block diagram for the combination. Card-size controller "AKI-H8" contains 32bits-CPU, A/D, D/A, SIO, RAM, FlashEEPROM and many ports. Electrode channels (max:8ch) are multiplexed for A/D and D/A. "High impedance of separation, high input voltage" analog switch is controlled by the CPU ports synchronizing with the system. more natural reaction by generating an appropriate reaction also to the EMG information that it does not learn in "execution" stage by tuning up so that the performer can get the feedback pulse according to his/her EMG information. The 32bits-CPU is powerful enough to construct this live NN parameter-mapping algorithm. E El Figure 19. Timing diagram of the system 5. Conclusions Some researches and experimental applications of humancomputer interaction in media arts were reported. New generation of EMG-sensor type instruments begins. I will continue the experiments and developments, and report the results in the future with some experimental performances. 6. References [1] Nagashima.Y., An Experiment of Real-Time Control for "Psuedo Granular" Synthesis, Proceedings of International Symposium on Musical Acoustics, (ISMA, 1992) [2] Nagashima.Y., Real-Time Control System for "Pseudo" Granulation, Proceedings of 1992 International Computer Music Conference (ICMA, 1992) [3] Nagashima.Y., PEGASUS-2: Real-Time Composing Environment with Chaotic Interaction Model, Proceedings of ICMC1993, (ICMA, 1993) [4] Nagashima.Y., Chaotic Interaction Model for Compositional Structure, Proceedings of IAKTA / LIST International Workshop on Knowledge Technology in the Arts, (International Association for Knowledge Technology in the Arts, 1993Aj [5] Nagashima.Y., Multimedia Interactive Art: System Design and Artistic Concept of Real-Time Performance with Computer Graphics and Computer Music, Proceedings of Sixth International Conference on Human-Computer Interaction, (ELSEVIER, 1995) [6] Nagashima.Y., A Compositional Environment with Interaction and Intersection between Musical Model and Graphical Model -- Listen to the Graphics, Watch the Music --, Proceedings of ICMC1995, (ICMA, 1995) [7] Nagashima.Y., Real-Time Interactive Performance with Computer Graphics and Computer Music, Proceedings of 7th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Man-Machina Systems, (IFAC, 1998) [8] Nagashima.Y., BioSensorFusion: New Interfaces for Interactive Multimedia Art, Proceedings of ICMC1998, (ICMA, 1998) [9] Nagashima.Y., 'It's SHO time' -- An Interactive Environment for SHO(Sheng) Performance, Proceedings of ICMC1999, (ICMA, 1999) [10] Nagashima.Y., Composition of "Visional Legend", Proceedings of International Workshop on "Human Supervision and Control in Engineering and Music", (IFAC,2001) [11] Nagashima.Y., "IMPROVISESSION-II": A Performing / Composing System for Improvisational Sessions with Networks, Proceedings of International Workshop on Entertainment Computing, (IWEC, 2002) [12] Nagashima.Y., Interactive Multi-Media Performance with Bio-Sensing and Bio-Feedback, Proceedings of International Conference on Audible Display, (ICAD, 2002) [13] Nagashima.Y., Interactive Multimedia Art with Biological Interfaces, Proceedings of 17th Congress of the International Association of Empirical Aesthetics, (IAEA, 2002) [14] Nagashima.Y., Bio-Sensing Systems and Bio-Feedback Systems for Interactive Media Arts, Proceedings of 3rd Interactive Media Arts, Proceedings of 3rd International Conference on New Interfaces for Musical Expression, (NIME, 2003) Figure 18. Block diagram of the combined system 4.2. Parameter control with NN Figure 19 shows one example of timing diagram in the development of time-share system. In this figure, the "EMG sensing" phase is set in common time-slot for 8 channels. There needs 2 special intervals: (1) bio-feedback setting time for switching the electrodes to protect the sensing circuits from the high voltage pulses, (2) muscle recovery time for EMG sensing from the shock of feedback pulses. Each bio-feedback pulses are set into separated time-slots from a viewpoint of medical dangerous prevention. There are many control parameters: (1) amplitude of pulse (OV-160V 7bits linear), (2) polarity of pulse, (3) width of pulse (0.2msec-4msec 7bits exponential), (4) interval of pulses (lHz-200Hz 7bits exponential), (5) pulse waveform (8bits * 8bits RAM programmable), (6) setting time from EMG sensing to pulse generation, and (7) recover time from pulses to EMG sensing. The NN(Neural Network) is one of the optimal solutions for making a compromise in the complicated relation of many parameters [1-2]. In "study" stage, a system can realize a