Page  198 ï~~Koji Kusu, Masaoki Computer Noh System Ino, Donguk Shin, Shuji Hashimoto, Sadamu Ohteru Department of Applied Physics Waseda University 3-4-1 Okubo, Shinjuku-ku Tokyo, 169 JAPAN and Koyo Honda Konparu School ABSTRACT Noh is a traditional Japanese musical play in which masked actors sing and dance accompanied by special instruments and a chorus. No attempt has ever been reported to computerize Noh, since it greatly differs from western music and is not thought to be suited to modern electronics. This paper presents a computerize Noh system to demonstrate that such music technology can be applied to non-western musical plays to open a new computerized music frontier. This system consists of five parts; the song analyzer, the dance analyzer, the video disc controller, the accompaniment analyzer, and the robot. actor. The proposed system produces the virtual Noh space in which people can participate in Noh plays as singers, as actors and as players. 1. INTRODUCTION Noh has been a traditional Japanese music play since the 14th century. Even modern artists pay deep attention to Noh as one of the most simplified and impressive styles, of musical performance. in the world. Many studies, not only in this, country but also abroad, have already been reported[l)[2] on this matter. A face expresses or reflects most the human heart or thinking. Noh, by covering performer's face with an artificial mask, calls for audience imagination and heightens their expectations. The few movements of the head are always significant, for the slightest turn or tilt of the mask can change its expression. This could be realized through sophisticated coordination among performers and members of the audience. Indeed, at an early age there had already existed a close cooperation among them. Much of the audience in a Noh theater was composed of amateur performers: students of chant(Utai), dance(Mai), one of the drums (hip drum - Ohkawa, shoulder drum - Kozutsumi, stick drum - Taiko) or the bamboo flute (Nohkan). Therefore it was possible for them to have a common knowledge base on the Noh system. However, only a few in the have such a knowledge base, because of the difficulties for obtaining the patterns or skills to play the instruments. As the result, many of them are rarely aware of the system behind the performance. It drives the audience.away from the Noh theater. Under these circumstances, using the recent virtual space technique we have attempted to construct a computerized Noh system In which people can participate in a Noh play without any special knowledge. 2. INPUT AND OUTPUT DEVICES and BASIC EXPERIMENTS 2.1 Input Devices Noh consists of chant, chorus, ensemble and dance. People could participate in computer Noh system as singers, as players and as actors. In order to realize to do this, we prepare a song analyzer, the accompaniment analyzer and dance analyzer as shown in Fig.l. The song analyzer detects the breath of a song voice in real time. The musical accompaniment analyzer recognizes the player's gestures by the help of a Data-Glove. It determines the instrument type, whether hip drum, shoulder drum, stick drum or bamboo 198

Page  199 ï~~- T0MCROPHONE P ITCII -TO -MIDI - SONG SONG CONVI.RTER ANALYZER VIRTUAL INSTRUMENT ACCOMPANIMENT - - ROBOT ANALYZER ERACTOR DAT CCD CAMERA ---REAL-TIME PATTERN DANCE LED [ POSITION DETECTOR ANALYZER VIDEO DISC CONTROLLER VIDEO DISC [ SAMPLE R 4,v MONITOR }[SPEAKER],.:~ Fig. 1 System Diagram flute, at its playing time. The dance analyzer with a CCD camera or Data Glove detects the actor's movements. CCD camera data are composed of the dance floor pattern (the trajectory of the actor's whole body movements) are used to control the video disc. Data Glove data composed of time varying 3-D coordinates of the actor's head, mask and hand movements are used to control a robot. 2.2 Output Devices The display or output device for Noh scenes Is very Important In order for the participant to obtain the high quality of virtual reality. A robot and video disc are used as the output devices. The former simulates the actor's body movements in 3-D space, and therefore It is best as the artificial output device. However, it is expensive. Furthermore, for more precise movement, the more control parameters it requests. Therefore, as a first step, we prepare a five degree of freedom manipulator (Mitsubishi Electronic Co. RV-M2) to be used as an output device with a simple hand-made actuator. They are controlled through the centronics interface. The video disc is soft-oriented, easy to handle, and response rapidly, although synchronization between scene and sound is difficult. The following scenes are prerecorded on the video disc: 1) The dancing figure of a professional actor masked and clothed in a stiff and varicolored costume on the stage. 2) The actor clothed in a soft and black crested kimono with hakama(divided skirts), the style of which is named "Hakama Noh". 3) The actor with his feet conspicuously clothed in white socks and being slid across the floor. 4) Scenes seen through the eye-holes of his mask during his play. Either of the above four scenes is switched improvisationally during a play, or two of them can be selected at will simultaneously ona CRT. 5) Accompaniment Scenes with Music Scenes of playing the drum, shoulder drum, stick drum and bamboo flute with vocal music. Their individual tone is input through the sampler and it can be selected with high, middle and low levels. 6) A variety of Mask Pictures(Graphics) They can change the shape of their mouths Fig.2 Facial Image Generated by Spring Frame Model (SFM) 199

Page  200 ï~~and also their expressions according to the tempo using a facial computer graphic based on the spring frame model(SFI) one of which is shown in Fig. 2[33. 2.3 Basic Experiments and Their Results Using the above mentioned input and output devices some basic experiments were carried out, especially with regard to interactive communication between the system and audience. In an actual Noh play, seemingly unrelated rhythmic and melodic patterns coalesce Into an organic whole, although there is no conductor In the system. Therefore, in computer Noh system, their exact timing is most Important. 1) A professional actor sings the chant. The system detects the breath of his song voice through a microphone, and controls the performer's video disc scenes according to his song's tempo. The breath interrupted periods (a break in singing style) which appear almost every seven beats were successfully detected in real time through the use of a high speed Digital Signal Processlng(DSP) unit. Some tempo detection errors were found when the actor moved his face against the microphone during his singing. 2) We recorded performer's video disc scenes: One of the authors(a professional performer) masked and clothed In a varicolored costume (or In a black kimono with hakama) enters the stage along the bridge, silently, his white-socked feet sliding over the surface, his heels barely leaving the ground while his toes flex upward to mark the completion of each step. Soon he begins to dance around the stage, gliding along the highly polished cypress boards, straight backed, leaning slightly forward, poised in potential movement even when standing still. Next, a participant in the audience(a student) with a small infrared light attached to his head walked to the right or left and forward or backward. According to his movements, a professional actor performed a play on the stage in a CRT display. 3) We also prerecorded musical accompaniment scenes: the hip drum, shoulder drum, stick drum and bamboo flute played by professional players through a video camera and voice sampler which can be controlled by MIDI. The drummers' calls "yao', "yaa" e te, which are the patterns of matching syllables to the count, were also entered Into the computer. Next we tried an experiment to see whether the above four instruments could be selected through an appropriate hand gesture or not. After learning the gestures for pIlaying instruments through the Data Glove, the system easily recognized the player's gestures and generated each instrument's sound while displaying its picture in real time. Moreover, we added a simple switching device to Data Glove and made a modified Data Glove, which we called an "Electric Drum". It could obtain low, middle and high level tones according to the positions on the drum where the player beat. 4) We prepared a Kyogen mask. Its modified facial Images - speaking, laughing, and crying expressions - were composed automatically after a texture mapping of the original mask Image by the use of the SFM technique, and we succeeded in making it change or react according to the actor's lines. 5) The robot composed of five degree of freedom manipulators was also masked and clothed in a bulk costume laced with gold. It has hands, but no legs, and therefore stands still as shown In FIg.3. At first, a professional actor performed a play or danced singing the chant, and the movements of his mask were entered through the Data Glove and those of his hand through the hand-made actuator into the computer. At the Fig. 3 Robot Actor 200

Page  201 ï~~same time, the tempo of his vocal music was detected through his breath in his singing voice. Next, if he sang the same chant once again, but with improvisation, the robot was able to control its play according to his singing tempo. With the same procedure, even if the actor dances under the accompaniment of the music, the robot can perform a supporting play in cooperation with him by matching its tempo to the accompaniment. 3. DEMONSTRATION The above successful experiments led the following demonstration at Waseda University on March 3, 1992, in front of an audience of more than two hundreds. 1) The Noh play "Awoi no Uye" One of the present authors, a professional actor sang the chant "Awoi no Uye". the system detected the breath of his song voice in order to control the video disc recorder to replay the "Awoi no Uye" performance scenes in real time, according to the song's tempo. The other author, a student, used gesture through the modified Data Glove, "Electric Drum", to play in time the virtual accompaniment instruments, selected improvisationally. 2) Kyogen "Kaminari" Kyogen, a traditional farce, is a genre of comic play which is performed between Noh plays by Kyogen actors, usually without using masks. Kaminari is typical, but one of the few comic play using a mask. We tried to perform the Kyogen with a new Kyogen actor, computerized live mask. The live mask moved Its mouth, and laughed or cried according to the actor's lines on the stage. It drew the whole audience into fits of laughter. 3) The Noh play "Hashi Benkei" This Is a famous ancient drama in Japan: Ushiwaka-Maru, a pretty young boy, stands on the Gojo bridge in Kyoto. Benkel, a ferocious Samurai, makes a cut at Ushiwaka with his long-handled sword. Ushiwaka jumps and trifles with Benkei by his short sword. One of the present authors made up as Benkei and a robot as Ushiwaka performed the play using the tempo of the chant "Hashi Benkei". 4. CONCLUSION Noh is a purely Japanese style performance art, It differs greatly from western music and has not been thought to be suited to modern electronic performance. Indeed, no conductor or music score has been made in western style for the system. Sometimes, the timing appears irregular, although it is exact as a whole. The proposed system provides a new way for the audience to enjoy and participate in Noh at any place and at any time, through the help of virtual reality techniques, without special knowledge of Noh performance. Some distinctive features are listed below: 1) The performance can be seen at a participant's will either with the bulk costume or with a "hakama". Note that people can see the movements better when the actors wear no upper dress. Moreover, they can also see the scenes through the eye-hole of the actor's mask during the dancing. 2) The robot actor and "Electric drum" can also be used In beginner exercises for Noh performance. 3) Expression control for Noh mask which Is essentially unchanged provides interesting problems from the stand point of not only drama but also human facial expression. 5. ACKNOWLEDGEMENTS We would like to thank Mr.A.Nakaya(Flute), Mr. A. Ko(Shoulder Drum), Mr. S. Ohkura(Hlip Drum) and Mr. K.Konparu(Stick Drum) for their cooperation. As for the English expressions of Noh performance in this paper, we owed much to the referencesi]. We also thank its authors. 6. REFERENCES [1] M.Bethe & K. Brazell: No As Performance, Cornell University East Asia Papers, No.16, Ithaca, New York, 1989 [2] E.Pound & Fenollosa: The Classic Noh Theatre of Japan. Greenwood Press, Publisher, Westport, Connecticut, 1977 [3] A. Sato, T.Harada, S. Hashimoto, S. Ohteru: Singing & Playing in Musical Virtual Space, Proc. of ICMC, pp.239-292 Montreal, 1991 201