Page  00000001 Physiological Measurement of Performers' Tension and its Utilization for M~edia Control Haruhiro Katayose (1)(2) Shigeyuki Hirai (1) Tsutomu Kanamori (1) Hirokazu Kato (3) Seiji Inokuchi (1)(3) (1) L.IS.T. (2) Faculty of Systems Engineering, Wakayama University (3) Faculty of Engineering Science, Osaka University (1) Shinsenri-nishimachi, Toyonaka, Osaka, 565-0083 JAPAN (1), Abstract In performing art, in addition to contents, performers' tension or elaboration is transferred to the audience. This plays a very important role in forming artistic impression. The authors have been engaged in producing interactive art utilizing gesture sensors. Using the sensing unit originally designed for gesture sensors, we have measured a performer's physiological data as the index of tension. We also measured and investigated concerning how the tension is conveyed to the audience using physiological sensors. This paper describes results of the experiments and discusses the effectivity of tension and physiological data in the performing art. 1. Introduction Needless to say, computer music has been growing with computer technology. Current evolution of computer technology enables us to perform live computer music, which could exist only as a tape piece few years ago. We have been conscious of music as the performing art again, the essence of music as it used be. The performer's tension or elaboration is conveyed directly to the audience in the performing arts. This plays a very important role in forming artistic impression. The goal of this study is investigating tension or elaboration around the interactive computer music. There is an interesting study done by M. Senju, a famous violinist with her colleague [Senju]. She investigated "how the performer's idea is conveyed to the audience." The procedure of the evaluation was based on questionnaires using adjectives. The point of our research is to consider the problem from the both views of physiology and psychology. There are some studies about measurement of stress or tension using physiological data in the field of ergonomics [Andreassi] [Shimono]. The basis of these study is that animals including human being are under the control of the sympathetic and parasympathetic nervous system, and this nervous system is relevant to stress or tension. Our approach of tension measurement is based on the same idea. One of the problems of physiological measurement is that the sensing system limits the subject's movement and it may influence the experiment accuracy. We prepared a wireless and compact measurement system to reduce this problem. This paper takes up two experiments regarding tension measurement. One is the analysis of a performer's tension in the different situations. The other experiment is to investigate how the tension is conveyed to audience. Next, this paper will discuss the use of physiological data regarding tensions as controllers in interactive art. 2. Tension and Physiological Measure There are various emotions which are rendered into performance. Above all, tense or excitative is a special emotion strongly linked to instinct, which is indispensable for animals to survive. This tense or excitative is under the control of a sympathetic nerve. On the contrary calm or sedative is under the control of parasympathetic nervous system. Physiological indexes generally used regarding "tense" are heart rate, skin potential and breath rate. They are indexes of the instrument well-known as Polygraph. Recently, thermography has been used to measure the face temperature. Heart rate is obtained from BEG or pulse and the scale called beat per minute (BPM) is used. Instant BPM increases in the tension state. The fluctuations of BPM~ are also indexes of tension. Fluctuation between 0.2 Hz and 0.3 Hz is mainly under the control of the parasympathetic nervous system. Fluctuation between 0.08 Hz and 0.1I Hz is under the control of the sympathetic nervous system [Iwanaga]. Skin potential has relevance to perspiration caused by tension. It is generally measured as a voltage between a palm and an earlobe or a wrist where the electric potential is stable. The decrease of the DC component ( under 0.07 8 Hz, SPDC afterwards ) and the increases of transitory SPR (Skin Potential Response) are the measure of tension. In this paper high frequency component of Skin Potentials above 0.078 Hz is used as the SPR index.

Page  00000002 3. Experiment Overview In this paper, we are going to deal with two kinds of physiological measurement, taking up a piece of interactive computer music called "Tikukan no uchu V" composed and performed by Satoshi Shimura [Katayose]. One is the analysis of a performer's tension in different situations. Tensions in practice, rehearsal and at the concert were measured. The other experiment is to investigate how the tension is conveyed to audience. It is examined from physiological and psychological (interview) data. 3.1 Tikukan no ucyu (Cosmology of bamboo flute) Thikukan no uchu (Cosmology of bamboo flute) is an interactive multimedia piece featuring CyberShakuhachi, which can electrically capture traditional techniques and manners of the shakuhachi performance. Thikukan no uchu consists of ten scenes. Each scene has each theme to construct the whole piece. Tikukan no uchu III and Tikukan no uchu V were performed at ICMC'94 and ICMC'96 respectively. Figure 1 is one shot from the concert on 18th March 1998. At the concert the physiological data were measured. The right lower image shows the data of pulse and Skin Potential. These data were also shown to the audience as a part of the multimedia piece. 3.2 Physiological Sensing Environment Figure 2 shows sensors for physiological measurement. These sensors detect pulse at an earlobe and skin potential. The data were transmitted through wired or wireless system. The obtained data are analyzed and the physiological indexes described in the previous section are extracted. Figure 1: Performance of Tikukan no ucyu Figure 2: Physiological Sensors and Transmitters 4. Experiment 4.1 Fundamental Experiment Figure 3 shows the physiological indexes used in our experiments. It is impossible to assert the reliability of the value of indexes. But it seems to be reliable when we want to know the tendency. Figure 4 shows the average skin potential for a certain phrase. The fluctuation caused by muscular movement is around 1 mV. Therefore, the fluctuation of this data is not by muscular movement. We see the performer's tension begins to rise after the start signal is given, that is, just before the performance. It is sustained while the performance and decays gradually after the end of the performance. Start signal Start e n -40 Skin Potential(mV) Heart Rate(bpm) SP.DC index SPR index 75.0 -50 1.5 70.0 - 40 1.0 -35 65.0 30 0.5 60.0 20 0.0 -30 performance calm performance calm performance iiiiiiiliiiiiiiiiiiiiiiiiiliiiiiiiii Figure 3: Physiological Indexes at calm status and in the performance -25 0 20 40 60 80 100 120 Time(sec) Figure 4: Average skin potential for a certain phrase.

Page  00000003 4.2 Tension Measurement of a performer in different situations This is the experiment to investigate how a performer's tension varies in various situations. Figure 5 shows the performer's subjective evaluation regarding tension for this piece, and BPMs of the practice, rehearsal, and of the concert. This piece is improvised music. Each performance time is different. Especially the performer performs scene 3,4 and 8 freely, conversing with the machine. The composition of this piece has twin peaks according to the composer. Scene 4 is the first peak, and the most expressive part is located at scene 8. From the figure, we see BPM of the rehearsal and the concert are gradually increasing. On the other hand, BPM in the practice is relatively low and stable. It is impossible to obtain definitive conclusion, but the authors would like to explain these phenomena as follows. The performer, against his words, lifts his emotion still in rather quiet part and relieve it at the very last position in the scene 10. practice the performer's subjective mark BPM 1 2 3 4 5 ( 8 9 10 160 140 120 21 60 40 20 123456789 10____________ 0 Time (sec) rehearsal concert BPM 1 2 3 4 5 8 91( BPM 160 ^ _!_ __ ^ l^ 160 140 140 120 I 120 100 100 80 fc A^ ^^ A t ^ I 80 60 I 60 40 40 20 20 Time (sec) Time (sec) Figure 5: The performer's subjective mark for each scene and BPM in practice, rehearsal, and concert. 4.3 How the tension is conveyed to the audience Next experiment is to investigate how the tension is conveyed to the audience. Due to the limitation of the number of the sensor channel, we measured only SPs for audience of three subjects. Figure 6 shows the audience's SPR index and Figure 7 shows audience's subjective mark for the performance. Figure 8 is the performer's SPR index then. First of all, it is difficult to find correlation between the audience's SPR and the subjective mark. If I were to venture a view, the audience is in the tension at the beginning part, because this was the first listening for him. The tension is increasing toward the end of the music. SPR Iex 1 2 3 4 5 6 7 8 9 10 5-- 2.2-. i............. l... I.ll... 2 "-.................n....................................... 1 2 3 4 5 6 7 8 9 10 0 120 240 360 480 600 720 840 960 Figureiiiiii7:iisubjective.. mark.of. the. audience. Figure 6: SPR index of an audience Figure 7: subjective mark of the audience

Page  00000004 It is also difficult to find correlation between the SPR audience's SPR and the performer's SPR. One of the nex 1 3 6 8 10 3.0 reasons of this phenomenon may be the performer's |_ _ difficulty to keep tensions in the experiment as he dose..... i...... in the concert. The audience's SPR rather seems to have 2.0.............................................. correlation with the performer's BPM at the concert, ___ _______ while we do not have the data of the same index unfortunately. One of the rational explanation might be 1.0 i- - as follows. The audience listens to the performance with |- |||- - || curiosity and gradually synchronize to the performer's o.o0,,,,,ime~(sec) tension, which is supposed to be that of the concert. 0 120 240 360 480 600 720 840 960 5. Utilizing Physiological Sensors for Interactive Art Figure 8: SPR index of the performer In this section, we would like to discuss the possibility to use physiological data as a controller in interactive art. We can divide the usage of bio-signals in two big categories. One is to use the direct or projected value of the signals, and the other is to use the fundamental resource which is extracted from the signals. One of the examples which interested the author is that by computer music by a mathematician, in 1988 [Streitberg]. In the presentation, music controlled by the author's brain wave was shown, as an ultimate computer music by mathematics. It is very interesting for the author. It is, while, difficult for the audience, especially in case of interactive music, to find what is relevant to the music. The latter fundamental resource is directly relevant to tension. It seems to be promising to add big synchronisity in interactive music. One of the problems to use physiological data as a tension index is the time resolution. Figure 4 shows that it requires about a few seconds to judge the performer getting tense or not,and, ten seconds relieving the tension or not. Tension indexes began to rise if the performer only has intention before the actual performance. This characteristics can be used effectively in the interactive art. We also experimented how the tension index changed, when the subjects have the intention only. We obtained the same tendency that it takes much longer to relieve the tension. In the experiment we verified that the time which well-trained people getting tension is shorter than that of not-trained people, but the difference is not so big. There is a big difference in the time to relieve the tension. Here trained people mean experienced music performers and dancers. The biggest factor of time delay is that of mental controls. Another factor is the time constants of the signal transformation system. For example, decrease of the Skin Potential (Figure 4) is caused by natural drying, the order of which time constant is seconds. There is an interesting report regarding mind reading which analyzes brain waves and recognizes what the subject is thinking. This research utilizes the characteristic pattern of brain waves when human being recall concrete images. The combination this mind reading with fast response and physiological measurement seem to be very interesting approach in future interactive art. 6. Summery This paper has been presented two kinds of physiological measurement regarding tension index. One was measurement of a performer's tension in practice, rehearsal and the concert. The other experiment was concerning how the tension was conveyed to audience. Some of the results are very interesting. But it is not sufficient to reach the definitive theory at the present. We would like to continue experiments from now on. This paper also discussed the possibility to use physiological data regarding tension in the interactive art. We conceived that physiological data would yield very unique pieces which we have never seen. We would like to produce an artistic piece with the scientific study. Acknowledgment Authors would like to thank Satosi Shimura for his shakuhachi performance, and Hiroyuku Tsutada for his research support. References [Senju] M. Senju and K. Ohgushi: How are the players ideas conveyed to audience?, Music Perception, 4, pp.311 -323 (1987) [Andreassi] J. A. Andreassi: Psychological Physiology, Nakanishiya Shoten (1985 in Japanese) [Shimono] T. Shimono, M. Ohsuga et al.: Estimation os Stress using, heart beat, breath and blood pressure, Human Interface News&Report, Vol.11, No.2, pp.135-138 (1996 in Japanese) [Iwanaga] M. Iwanaga and M. Tsukamoto: Effects on Excitative and Sedative Music on Subjective and Physiological Relaxation, Perceptual and Motor Skills, 85, pp.287-296 (1997) [Katayose] H. Katayose, T. Kanamori, S. Simura and Seiji Inokuchi: Demonstration of Gesture Sensors for the Shakuhachi, Proc. ICMC, pp.196-199 (1994) [Streitberg] B. Streitberg and K. Balzar: The Sound of Mathematics, Proc. ICMC, pp.158-165. (1988)