Page  00000001 GDS (Global Delayed Session) Music --- new improvisational music with network latency Yoichi Nagashima * Takahiro Hara, Toshihiro Kimura, Yu Nishibori ** * Shizuoka University of Art and Culture / Art & Science Laboratory ** YAMAHA Corp. Abstract This is a research report of improvisational computer music with human-computer interaction and music education. Many sensors are used for the interactive communication as interfaces, and many algorithmic agents are connected via networks each other. This paper is intended as an investigation of some special approaches: (1) Unix(Irix)-based network session system called "Improvisession-I", (2) New music model called "GDS (global delayed session) Music" allowing heavy newtork latency, (3) PC-based network session system called "Improvisession-II", (4) Combination of many sensors and interfaces for free improvisation in music called "Improvisession-III" system, and (5) Application prototype produced by YAMAHA. 1. Introduction The research called PEGASUS project (Performing Environment of Granulation, Automata, Succession, and Unified-Synchronism) have produced many original systems and realized experimental performances. The next step of the research have produced a compositional environment with intersection and interaction between musical model and graphical model. The keyword of the project was "listen to the graphics, watch the music", and the purpose was constructing the composition/performance environment with multi-media (Fig 1). Scenario (ArtistU Composer) Soundlnput| I Sensor Videolnput Parameter Fusion Parameter Messages Control System (Time I Space; Structure) Messages "Sound World" Model "Visual World" Model (Sound Generate Agents) (Graphics Generate Agents) Acoustic Pixel Database Database Note Music Theory Object Visual Theory Level Music Psychology Level 3DModeling Visual Psychology Composition Composition Sound Outut Visual Output Figure 1. system of the PEGASUS project The system consists of some types of agents that are linked by object-oriented networks. The "control" agent exists in the center of the system. This agent generates the "control messages" both to the sound agency and the graphics agency in time domain, spatial domain, and structural domain. The input messages of this agency may be divided into four types: (1) traditional scenario of performers, (2) sensor information of the performance, (3) real-time sampled sounds as the material of sound synthesis, and (4) real-time recorded images as the material to generate graphics. The composer can describe the scenario as the algorithmic composition in a broad sense with both sound and graphics [1-12]. 2. Improvisession-I The first experimental system of music education was called "Improvisession-I". The system run with 24 SGI Indy computers through FDDI LAN, and softwares were developed with Open-GL, OSF/Motif, and RMCP (Remote Music Control Protocol) by C language. RMCP was produced by Dr. Masataka Goto [13], and the RMCP package is opened to the world on the Web. The system seems like "drawing game" screen, and players can use mouse, keyboard and MIDI instruments (Fig 2). Figure 2. screen shot of "Improvisession-I" All students can play their own phrases, and can broadcast their phrases to other players with improvisation via RMCP. In the experimental lectures of students of music course of Kobe Yamate College, there were many interesting results in the theme of entertainment and education. For example, a certain

Page  00000002 student liked inviting a specific partner and playing in private like a chat rather than participating in the music session in all the members. Or probably because it became that the phrase which student performs in session of all the members tends to be missed, there was a tendency for all the members' volume to increase gradually. 3. GDS music Arrangement of the problem about the research so far pointed out the following improving points. First, this software was interesting in musical education with some kinds of entertainment, but the required system is not popular with general environments in academic field. Secondly, the RMCP system required the "Time scheduling" methods which manages the synchronization with latency of networks, but both methods (RMCPtss [RMCP Time Synchronized Server] or NTP [Network Time Protocol]) seemed heavy for simple personal computers. And in order to absorb the possibility of the variation in delay by the network flexibly as the 3rd biggest subject, the necessity of reforming the concept of the music on condition of simultaneity itself has become clear. Dr. Goto also advocated the concept of "the music which was late for the difference" in research called Remoto-GIG [13]. However, the concept is extended further more flexibly and it came to build an idea called GDSmusic (Global Delayed Session music) here. network like chat and BBS, is expected further. 4. Improvisession-II After these experiments and researches, We have newly produced the second experimental environment and system called "Improvisession-II". The platform have changed from SGI Indy computer to Macintosh computer connected with Ethernet (Fig 4). timbre - -lli Figure 4. screen shot of "Improvisession-II" This software is developed on Max/MSP environment using OpenSoundControl developed by CNMAT [14]. The "otudp" is a UDP object for PPC using Apple's OpenTransport networking system. All students can play their own phrases, can play with chaotic/statistic generators and can communicate with other performers with improvisation via UDP/IP. As a difference from RMCP, the mechanism in which each user's time is managed in this system is unnecessary. Moreover, it is necessary to transmit no music performance information to real time like RMCP, and since what is necessary is to exchange only the parameter of the algorithm music generation which runs autonomously within each personal computer, the traffic burden of a network is mitigated. The important thing of the concrete composition elements of this system is introduced to below, and it considers as the help of an understanding of the new concept of GDSmusic. 4-1. Backing/Melody Generator In a base generator block, the pattern of many scale phrases permitted corresponding to the code is prepared as a table (Fig 5), it chooses statistically, and a performance is generated. In a Melody phrase generator block, students can perform not only by using musical keyboard graphically but also by controlling parameters which manage the selecting probability of each note in the musical scales. Figure 5. bass phrase generating tables 4-2. Synchronizing Management Player A 7 - -:::.::.;..;,;"........... 5 - i i;:......... r...... / ILe tl i)1 i..ii) 3i A Player B Player C Figure 3. concept of "GDS Music" Figure 3 shows the concept of "GDS Music". We made unnecessary time management which RMCP demanded, and the users by whom network connection was made enabled it to enjoy a music session in the framework of GDS Music, without caring about the delay according to each. Of course, for that, the concept of the simultaneity which was the traditional foundations of a music session is thrown away, and the new view of playing in musical session with "1-2 measures past shadow of a performance of the partner" is needed. The idea of this GDSmusic was specifically first realized as the system "Improvisession-II" introduced in the following. However, the new entertainment and new education of not only this but a network era, i.e., a possibility of saying that many and unspecified persons enjoy themselves through a

Page  00000003 In a transmitter block (Fig 6), selected BGM parameters and information of played melody phrases are packed for OpenSoundControl message and transmitted at the moment of the end point of the looping. As an important point, this transmission is free according to the timing of performance loop, without caring about the synchronization with other players who has participated in the network session performance. Figure 6. subpatch of message transmitter block The receiver block (Fig 7) is the important point of the synchronization mechanism in GDS Music. At first, the incoming OpenSoundControl message is latched at the moment of Ethernet communication. Secondly these latched information is latched again at the moment of the end point of the looping of receiver's performance. Finally these latched information is triggered with internal timing clock pulses to realize the ensemble with its internal musical performance. Consequently, the fellow participants who has not taken the synchronization of timing to each other can enjoy a music session with a performance of other participants who were late and arrived, forming each performance. improvisatorial music performance and media installations with sensors. Figure 8 shows the "Touch Sensor Board" developed as an installation work but used later as an interface (instrument) in computer music. Students can act freely with this, and the MIDI output is used as one of the input actions of Improvisession-III. It is interesting that students acts freely from the traditional music style in spite of they did not study about musical improvisation. This is considered to have an important meaning on musical education. Figure 8. touch sensor for Improvisession-III 6. Developent of Application Prototype We(YAMAHA Corp.) developed a musical session system over the network as an applied model of GDSM. The outline about this system is mentioned: (1) A player plays some electronical musical instrument connected to a terminal that is connected to some network. (2) A session could be realized from communicating musical data between the terminal and another terminal built in the same system. (3) Each terminal has some mechanism mentioned: (a) Each terminal joined the network session has an original clock function that has 1 cycle consisted of 4 measures, and a player plays with the clock count. (b) A musical data played by a player would be sent to another terminal with a "time stamp" which shows when it was played in the 4 measures. (c) Receiving musical data from other terminals, a terminal does not play back immediately. (d) It's going to play back when the position of its local clock corresponds to the time stamp attached to receiving data (Fig.9). Therefore, the musical data received from other terminals is played behind time. A player plays with 4 measures which other players performed before. (e) In a case of more then 2 players, a terminal does not send musical data to all other terminals directly. The data would be sent to others through a terminal connected to your terminal directly (Fig.10). (f) On the window of each terminal, you can see how the members joining the session connect one another ( Fig.11). (g) Moving around the map on the window, a player can change the connection conditoins. (h) Each terminal has a window that enables to see some received others' musical data (Fig.12). The vertical axis expresses time and the horizontal axis expresses pitch on the window. Musical Figure 7. subpatch of message receiver block 5. Improvisession-III Although Improvisession-II achieved success temporary as a musical education system, it was pointed out that the way of thinking is limited to the framework of traditional music theory as the greatest fault. Because it suited using the interface based on Western theories, such as a musical score and a music keyboard. Then, we experimented by developing new "Improvisession-III" which utilized various types of sensors developed as an

Page  00000004 data received from other players go down from the top of the window, and when it reaches to the bottom, the sound would be generated. terminal A, p termnal B Fig-re 9. playingpositions Figure 9. playing positions Figure 10. terminal connections sound. (3) A player can change the conditions of the connection as he wants, and if it's changed, the music would also be changed. (3a) Moving around the map on the window, you can change the connection conditions and also let a player join another session group. (3b) If the conditions of the connection would be changed, the music would also be changed because a further player's musical data would be received later. (4) A large-scale system is not needed - A server to synchronize timers that each terminal has and a system guaranteeing the delay are not needed at all. Therefore it is adaptable to various network environments, for example, the difference of connection and the quality of the network and so on. 7. Conclusions We described the developments of education systems for improvisational sessions with networks and a concept called "GDS (Global Delayed Session) Music". We will develop new sensors for this system as new human-computer interfaces and research new possibility in network education and entertainment in multimedia. 8. References [1] Nagashima.Y., An Experiment of Real-Time Control for "Psuedo Granular" Synthesis, Proceedings of International Symposium on Musical Acoustics, (ISMA, 1992) [2] Nagashima.Y., Real-Time Control System for "Pseudo" Granulation, Proceedings of 1992 International Computer Music Conference (ICMA, 1992) [3] Nagashima.Y., PEGASUS-2: Real-Time Composing Environment with Chaotic Interaction Model, Proceedings of ICMC1993, (ICMA, 1993) [4] Nagashima.Y., Chaotic Interaction Model for Compositional Structure, Proceedings of IAKTA / LIST International Workshop on Knowledge Technology in the Arts, (International Association for Knowledge Technology in the Arts, 1993Aj [5] Nagashima.Y., Multimedia Interactive Art: System Design and Artistic Concept of Real-Time Performance with Computer Graphics and Computer Music, Proceedings of Sixth International Conference on Human-Computer Interaction, (ELSEVIER, 1995) [6] Nagashima.Y., A Compositional Environment with Interaction and Intersection between Musical Model and Graphical Model -- Listen to the Graphics, Watch the Music --, Proceedings of ICMC1995, (ICMA, 1995) [7] Nagashima.Y., Real-Time Interactive Performance with Computer Graphics and Computer Music, Proceedings of 7th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Man-Machina Systems, (IFAC, 1998) [8] Nagashima.Y., BioSensorFusion: New Interfaces for Interactive Multimedia Art, Proceedings of ICMC1998, (ICMA, 1998) [9] Nagashima.Y., 'It's SHO time' -- An Interactive Environment for SHO(Sheng) Performance, Proceedings of ICMC1999, (ICMA, 1999) [10] Nagashima.Y., Composition of "Visional Legend", Proceedings of International Workshop on "Human Supervision and Control in Engineering and Music", (IFAC,2001) [11] Nagashima.Y., Interactive Multi-Media Performance with Bio-Sensing and Bio-Feedback, Proceedings of International Conference on Audible Display, (ICAD, 2002) [12] Nagashima.Y., Interactive Multimedia Art with Biological Interfaces, Proceedings of 17th Congress of the International Association of Empirical Aesthetics, (IAEA, 2002) [13] M.Goto et al. RMCP: Remote Music Control Protocol -- Design and Applications --, Proceedings of 1997 ICMC, (ICMA, 1997) [14] W.Matthew et al. OpenSound Control: A New Protocol for Communicating with Sound Synthesizers, Proceedings of 1997 ICMC, (ICMA, 1997) [15] Nagashima.Y., "IMPROVISESSION-II": A Performing / Composing System for Improvisational Sessions with Networks, Proceedings of International Workshop on Entertainment Computing, (IWEC, 2002) Figure 11. screen of the connections Figure 12. screen of terminals The vertical axis expresses time and the horizontal axis Some features about this system is mentioned: (1) Absorbing the delay of the network - It is possible to absorb the delay of the network by delaying a unit consisted of 4 measures. So the delay wouldn't ruin the music. (2) Enabling to see the musical data that you received. (2a) It is possible to see some musical data that you received from other players until the sound would be generated. (2b) It is difficult to ansemble in the condition that you're only listening to another player's sound. But because in this system you can see the sound as a ball, that would be generated later, you can have some margin to reply to another player's