Page  00000232 The Musical Session System with the Visual Interface Yuichi Yoshida, Yasunori Yamagishi, Kazuki Kanamori, Naoki Saiwaki, Shogo Nishida Department of systems and human science Graduate school of engineering science, Osaka University email: yuichi@nishilab.sys.es.osaka-u.ac.jp Abstract In recent years, several music composition systems have been published. They operate cooperatively with emotional input. However, most of these systems require users to have the ability to play music well. We have constructed a musical session system that children and novices can play pleasantly. The session system is composed of two parts. One is musical, a kind of instrument with which users can perform music by modifying a combination ofthe phrases. The other is a visual part which allows for visual, realtime, interactive manipulation during performance. This prototype was evaluated by many children. The result of the evaluations showed that children could play this system pleasantly. 1 Introduction Recently, due to developments in computer technology, automatic musical performance, cooperative performance, visualization of musical information and media arts have been explored in new ways. For instance, in research on session systems, a communication model between users is implemented in order to emulate a jazz style (Masataka Goto 1999), (Sanae Wake 1994). However, such systems demand that users should be skilled musicians. Now there are media arts with which users can enjoy sounds and pictures interactively as well as a new form of music score that is represented by utilizing 3D CG. Examples include the research of the visualization of music information (Kenji Katsumoto 1997). Most of interactive arts and media arts are constructed from artistic or other ambiguous factors. Most of these programs place little importance on the users' viewing of the music representative CG. In the research of the musical score, most of these systems visualize music that was already composed or played. Little research has been done in the field of the dynamic, real-time visualization of music. Based on the above points, we propose the musical session system combined with a visual interface. The visual interface can visualize the change of the condition of the performance in realtime. The visual interface displays multimodal pictures and adds enjoyable sounds. Further more, an input device independent from conventional instruments is added to the session system. We constructed and evaluated the prototype system as an integrated performance environment and session. 2 Musical Session System 2.1 Session System The existing session systems make phrases according to the music chord by analysing the structure of music. Therefore, users must have the ability to play music and proficiency in the existing systems as well as knowledge of music in order to perform with the session system. Our session system accounts for musical construction (Yasunori Yamagishi 2001). The session system makes musical performance by selecting previously provided phrases based on analysis of the combination of the music phrases. Users need not have the ability to play music, nor do they need knowledge of music because this session system enables users to perform session like modifications of music atmosphere by selecting prepared phrases freely like a DJ. 2.2 Visualization of Music We realize the ability of CG not only to visualize the sounds of the scores but also to enjoy the changing of sounds (Yuichi Yoshida 2001). However, most of the existing systems only represent the information of the music that was already played or composed as a score using CG. Our interface can visualize users' performance in real-time, as it is connected to the session system. We considered the following two points very important factors for constructing the visual interface of the session system. 1. How to visualize the information of music as a real-time session. 232

Page  00000233 The visual interface enables users to understand musical parameters, such as chords and velocity. The visual interface represents the sounds as components of the instrument that make sounds themselves. 2. How to embody a sense of playing with 3D objects that reflect musical parameters. Musical factors that can be operated by users must conform to users' abilities to perform and knowledge of music. Based on the above two points, we constructed a visual interface that can dynamically display states of performance in the new environment of musical performance. 3 System Architecture 3.1 Session System Considering Musical Construction prepared phrases \jJ\ Q I T computer(Mac)... ji )-... Ii ' sound source s o..........."o.B n send looing phrases send looping phrases send MIDI messages I-U----- Sisual parl musical part Figure 2: Architecture of the musical part '(P r,ý i p] tic MncinloshG4( I * I,so!i1d SOilircc \ isiuil iiinterface Iusers I t.lgcZl'S iIIcrhicc.s Figure 1: Architecture of prototype system Figure 1 shows the architecture of the prototype session system. The visual part is composed of a graphic workstation (Windows2000). The musical part is composed of a Macintosh G4, a sound source (XV5080, Roland), and three input device interfaces. The graphic workstation and Macintosh G4 communicate through MIDI. The Macintosh G4 receives data from the interface, processes and sends the performance data to the graphic workstation and the sound source. Based on data from Macintosh G4, the graphic workstation displays the visual interface while the sound source plays music of the session. Figure 2 shows the architecture of the musical part. The computer manipulates previously prepared looping phrases and plays music by sending the phrases to the sound source. Users can play music by selecting looping phrases. The selected looping phrases are sent as MIDI messages to the computer. There are different kinds of the looping phrase. Each phrase has three atmospheres such as lively, a little lively, and calm. Users can also modify these phrase parameters. 3.2 Structure of Graphical Interface Figure 3 shows an example of a scene provided by the visual interface. The visual interface displays some objects and the scene. An object is a deformed cactus based on the image of the music preformed by the session system. Each cactus is animated with an instrument. The cactuses' dance is changed by the phrases the users have performed. Users can confirm their own currently playing phrase by way of the CG. Further more, users can pleasantly understand the difference between their own performance and another user's performance at a glance. The visual part receives the MIDI message as a parameter from the musical part. The parameter from the musical part corresponds to the atmosphere of the performance as each user plays, the distinction of the phrases, and so on. The visual interface displays the state of each instrument's performance based on these parameters. The sum of the whole musical atmosphere is represented as the whole picture. 233

Page  00000234 keyboard arm sensor Aso drums 1117101 Notice too that the number of the examinees who played the drums is higher than the number who played the keyboard. Table 1: Examinees the group of age number kindergarteners (under 5) 31 low primary schoolchildren(under 8) 131 high primary schoolchildren(under 11) 95 junior high school students(over 12) 10 100 80 music scene Figure 3: interface Outline and screenshot of the visual 60 40 20 0 U drum [Ikey 3.3 Input Device with Arm Sensor Our systems use instruments as the interface to the systems. In this system, a bending angle sensor is added to the interface so that the human body itself is used as an interface. Therefore, this interface is independent of the existing instrument interfaces such as drum, piano and etc. We designed an interface that senses the bending degrees of the elbow. This interface enables users to perform music by moving their arm as if playing the guitar. 4 Experiments 4.1 Condition of Experiments In summer 2001, we exhibited this prototype system at "Hukuzaki Virtual Wakuwaku Hiroba" (the entertainment for children with virtual reality) in Osaka, Japan and got a lot of children to play this prototype system. Table 1 shows the examinees that evaluated this system. The procedure for the experiment is below. At first two or three children selected an instrument to play, and we simply taught them about our system. Then the children played our system freely. We shot a movie while they were playing with the system. We also got them to answer an evaluation form after playing. The evaluations were done by observing digital video and reading the evaluation forms. 4.2 Results Figure 4 shows the rate of examinees that understood the response of the system, classifying into four age groups, and two instruments. Notice the age, young kindergarteners and low primary schoolchildren can understand the system's responses. Older examinees understand the response better. under 5 under 8 under 11 the groups of age over 12 Figure 4: Rate of children who understood the response Figure 5 shows the number of examinees who played rhythm, classified into four age groups, and two instruments. There were more examinees that played rhythm than examinees that played the keyboard. We suggest that it is more difficult for examinees to play the keyboard than drums because the keys of the keyboard are small. Figure 6 shows the number of examinees that played while watching the visual interface. In any groups, examinees who watched the visual interface among children who played the drums were more than children who played the keyboards. As examinees become older, the number of the children who watched the visual interface tended to be higher. Because it is more of an imperative to watch your hands while playing the keyboard, it is difficult for young children to simultaneously watch the visual interface and playing instruments. 234

Page  00000235 100....................................................................e n c es........................................................... R e f e r e n c e s Kenji Katsumoto, Naoki Saiwaki, S. N. (1997). 80 Construction of the music score by using 3d cg. Information Processing So-cietyof Japan, 587-588. 60 U drum Masataka Goto, Isao Hidaka, H. M. Y. K. Y. M. (1999). A Oi key virtual jazz session system: Virja session. Information 40 B - - Processing Society of Japan 40(4), 1910-1921. 20Sanae Wake, Kouichi Kato, N. S. S. I. (1994). Cooperative musi-cal partner system using tension-parameter: Jasper(jam session system). Information Processing o Society of Japan 35(7),1469-1481. under 5 under 8 under 11 overl2 SYasunori Yamagishi, Naoki Saiwaki, S. N. (2001). the the groups of age. cooper-ative session system noticing the construction of Figure 5: Rate of children who played on rhythm the music. In Proceedings of the 45th Annual Conference of the ISCIE, pp. 169-170. Institute of Systems, Control and Information Engineers. 100 Yuichi Yoshida, Naoki Saiwaki, S. N. (2001). Construction of the visual interface on the session system. In 80 0 Proceedings of the 45th Annual Conference of the ISCIE, pp. 163-164. Institute of Systems, Control and 0o Information Engineers. U drum C key 40 20 0 under 5 under 8 under 11 over 12 the groups of age Figure 6: Rate of children watching visual interface 5 Conclusion In our research, we constructed and evaluated the prototype music composition session system for children and novices. In the evaluation, the effectiveness of the musical part shows that children could perform pleasantly by selecting prepared phrases. The visual interface deepens children's ability to pleasantly perform and comprehend a music session. We propose that the prototype system be applied to the educational systems for children and the rehabilitation system for the elderly and handicapped persons. We are examining how the prototype system can be applied to an educational program. Such a program would have music, rhythm, and other courses. For example, the system could make children dance or play music on rhythm through utilization of the visual interface. 235