Page  00000118 A MUSICAL FRAMEWORK WITH SWARMING ROBOTS Yuta Uozumi Media Design Program Graduate School of Media and Governance, Keio University SFC, Japan isanal 37@sfc.keio.ac.jp ABSTRACT Masato Takahashi Media Design Program Graduate School of Media and Governance, Keio University SFC, Japan masatooo@sfc.keio.ac.jp Ryoho Kobayashi Media Design Program Graduate School of Media and Governance, Keio University SFC, Japan ryoho@sfc.keio.ac.jp In this paper, we describe an approach to a musical framework with interactions among numerous physical autonomous devices. The devices behave as metaphors of life and self-organize sounds and rhythm. Users can manipulate the system by affecting the interactions of the devices. We implemented the system as two different installations. 1. INTRODUCTION This system is implemented as a sound-installation, which generates sounds through interactions among swarm robots. Various models for swarm robots have already been proposed [1]. Almost all of them have been focused on how to play musical instruments using robots. However, the model proposed here is unique in that it is focused on the generation of musical structures through the interactions of autonomous, swarming robots [2]. The robots are called "agent-robots". The agent-robots behave as metaphors of actual lives. They were developed in the image of insects. They have insect-like shapes and, like insects, they swarm to seek food. Therefore each agent has a microcontroller and sensors mounted on it. The agents are programmed to seek, chase and eat food. Their food is light, which they seek with their sensors. If they find light, they move to eat it. An LED flashes on the agent's body when it is eating. The LED's color and position are tracked by a CCD-camera that is mounted on the overhead. The sound generator on an external PC generates sounds based on the information. Users can give the agents light as food. When users place red paste or red LEGO blocks on a scan-board, these shapes of those items are displayed as LCD light below the agents. The agents eat the light when they find it. They also interact with each other. In response, the agents self-organize [3] rhythm and pitch. Six agentrobots are employed in this system'. 2. SYSTEM The system consists of four sections as follows (Figure 1): A: Insect type agent robot B: Human interface C: Tracking system with overhead camera D: Sound generator 2.1. Insect type agent robot The agent robot is mounted a microcontroller and sensors. It is programmed to seek light. Light is food for the agents. Agents react to light on LCD (See Fig.I1 -B 1) and in the environment. If an agent finds a light, it chases the light to eat. When the agent chases a light, an LED on its body flashes its predefined color. The LED color and position are utilized in the tracking system (See Fig. 1-C). The tracking system is described later. Six agents are employed in this installation. 2.2. Human interface A scan-board is set in front of the user (See Fig.1-B3)2. The board is captured by a USB camera (See Fig.I1 -B2)3. A user places red paste or red LEGO blocks on the board, and then the shapes are expressed below the agents as LCD light (See Fig. 1-B 1). If the agents locate the light, they chase it to eat it. During this process, these insect type robots repeat flashing their LEDs. The flashes are utilized for generation of rhythm and pitch. 2.3. Tracking system with an overhead camera The overhead camera tracks the movements of the agents. It detects the color and position of the LEDs flashed by the agents. The tracking system sends the detected information to a sound generator (See Fig. 1-D). 2.4. Sound generator The sound generator plays sounds in accordance with signals from the tracking system (see above section 2.3). The agents have sexuality. Male agents emit LEDs with blue color when they are eating light. On the contrary, female agents emit LEDs with red color when eating. The blue lights are utilized for rhythm. The sight, which is viewed through an overhead camera, is separated to multiple grids by the system. The sound generator plays predefined sounds based on the grid where the blue flash is detected. If the system detects a red flash, it plays low-frequency sounds depending on x-axis value of the position. An agent has a built-in 1Please refer the demo movie online. http://www.mag.keio.ac.jpl~isanal37/bd/Demo.html 2The scan-board size: Width: 300mm Depth: 2110mm 3 The LCD size: Width: 930mm Depth: 520mm 118

Page  00000119 contact microphone. It plays physical noises of motor and scratch. These sounds are processed as effect source by the sound generator. mmmmmml -m-m-m-m-m-m-m-m ---------------------- ýxxxx:,.................................................................................................................................................................................................................................................................................................................. System Diagram A: Bg tpe aqgnt nrots SBl: LCD B-: uS am -s scan oere a C,; Qvr I 4nad ram,ý r ar E. LED Each agent-robot has a LED. It can emit three colors simultaneously. According to the combination of the three colors, it generates multiple colors. An agent flashes the LED with red or green or blue color based on the agent's condition for generating sounds. It normally flashes an LED with green color. A. Micro Controller D Motors Alma"a~ Coimputer t.{ppi~ l al tG) Fig're 1. System Overview Figure 1. System Overview Figure 2. Architecture of an agent-robot 3. AGENT-ROBOT The agent-robot is the most important component of the system. Commonly, most devices for sound control run individually and statically. However in the proposed system, the physical components swarm and interact with each other as agents. In this section, we see the implementation of the agent-robots. 3.1. Architecture of an agent-robot An agent-robot consists of the following components (see Fig.2). A. Micro Controller Each agent-robot has a micro controller for autonomous control. The controller can process multiple outputs and inputs, which are digital or analog signals. It is made by AVR Inc. Plural sensors and motors are connected to it. B. Cadmium Sulfide (CDS) sensor Two CDS sensors are mounted on each agent-robot. They detect the intensity of lights on the left and right in front of each agent. The difference of light intensities between left and right decides the direction of each agent's movement. C. Infrared (IR) sensor An IR sensor is employed to detect barriers in front of each agent. If the sensor detects barriers 2 or 3 centimeters ahead, it notifies the Micro Controller. D. Motor Two motors are incorporated into each agent-robot for its moving. The drive power of each motor is decided based on the difference between the values of the two CDS sensors. 3.2. Algorithm of the agent-robots The algorithm of the agent robots is as follows. 1. An agent sets the light intensity of the environment as initial value when the power is on. 2. At first, the agent moves slowly to seek light. The direction of movement changes at random. 3. If the agent detects light intensity larger than the initial light value, it moves in that direction. 4. When the agent loses track of the light intensity, it goes back slightly because it has passed over the light below. Through the repetition of this process, eating behavior emerges. 5. If an agent detects a barrier, it changes direction of movement to the left or right at random. The algorithm is simple, however, complex interactions emerge from it. They are behaviors like scrambling for foods or yielding them according to the user's feed. 4. RESULTS OF THE FIRST IMPLEMENTATION A lot of physical interfaces, which correspond to user's input directly, already exist. These linear interfaces often run statically. However, this project approached an interface model by which the user affects interactions between plural physical autonomous devices. This interface is dynamic. If we adopt a new specification for a model like this, anyone can put in a newly designed agent based on the specification. Then the interface's behavior might change dramatically. 119

Page  00000120 The project has many future works such as how to balance between control and out of control, how to stimulate meaningful self-organization and how to develop more optimized models for musical purposes. 5. ATTEMPT OF AUTONOMOUS MUSICAL GENERATION FOR AN INSTALLATION In addition, we attempt to implement a system that generates musical patterns autonomously based on the proposed framework. The system was updated for it. It was implemented as a next step of our installationproject, named "musicalBox"'. 5.1. Concepts The idea of the implementation was inspired by a concept of autopoiesis (Maturana, Varela, 1980) [4]. It was designed to exhibit as an installation. An autopoietic system continues to behave evolutionarily, even if there is no external signal. This is because it can generate signals by itself. In the proposed system, the input was changed from feed by users to environmental sound. This generates new musical-evolutions through agent-robots' interactions according to their own self-organized and/or environmental sounds. The generated sounds brew up the next interactions of agent-robots. In such a system which includes feedback, nonlinear components are needed to obtain interesting behaviors. Therefore, the sound component of a system is changed from digital sound-synthesis to playing the physical piano. This can enhance the dynamic behavior of the system because such feedbacks from physical-components take in errors and environmental noises or reverberation of the space. In addition, agent-robots are mounted a communication system to aid of interactions among agent-robots. 5.2. System Overview In this section, we see embodiment of the system. the detection with predefined rules. Consequently, the flashes are transformed to piano playing. The lights below the agent-robots and piano playing form recursive-loops via the interactions of the agents. I -C'tiip a. I Lights on he LCD a.......... I [...... Sack ssstem on Figure 3. Diagram of second implementation 5.2.2. Implementation of Playing the Piano Eight notes are utilized to play the piano. These notes are selected carefully to accord with the atmosphere of installation by the author (see Fig.4). Therefore, when the notes are played asynchronously by the agent-robots, the sounds can keep the image of the installation. The keyboards corresponding to each note are assigned actuators (Section 5.3.2), and are pressed by the system with the actuators. Two types agent-robots, male and female, exist. They flash LEDs based on their sex and predefined algorithm (Section 2.4). If the male agentrobot flashes a blue light, from its LED, a note is selected according to the position, and is played. When the female agent-robot flashes a red light from its LED, a simple motif consisting of the eight notes is played according to the detected position. In consequence, the system generates music which has a fluctuation structure. 3 --- -- - L J,, _1 Figure 4. Selected notes 5.2.1. Feedback System Interaction source for agents is changed from feed via users to feedback of environmental sounds (see Fig.3). The feedback system is implemented to always analyze the pitch and amplitude of the environmental sounds. If the system detects a sound which has certain amplitude, the system displays lights below the agent-robots, based on the frequency and amplitude of the detected sound. An agent-robot flashes a LED on its body when eating the lights (Section 3.1E). The flashes are detected by the system. The system drives piano-actuators according to 1 Please refer to the demo to observe agents' behavior of this implementation. http://www.mag.keio.ac.jp/~isana3 7/dock/icmcO7/ 120

Page  00000121 5.3. Added Components of the System Added components for the autonomous system are as follows: 5.3. 1. robots Communication System of agent Agent-robots are modified to communicate with each other. The communication system was implemented with infrared radiation (IR). Agent-robots are programmed to chase away neighbor agent-robots of the same sex when detected. On the contrary, when the neighbor agent-robot encounters one of the opposite sex, it performs courtship dance. If other agents come to the pair of agents, they perform the dance too. Consequently, the courtship dance has contagiousness. The communication algorithm is as follows: 1. An agent-robot always transmits signal code-"A", which indicates its existence. 2. If another agent detects the signal code-A, it transmits signal code-"B", which indicates its own sex. 3. The agent-robot which received code-B, compares its sex with the other. In the same sex case, it performs intimidation behaviors and transmits intimidation signal codes (signal code-"C1"). In the opposite sex case, it performs a courtship dance and transmits courtship-signal codes (signal code-"C2"). 4. The other agent which received code-C 1 (intimidation-signal) backs away to escape. On the contrary, if it receives code-C2 (courtship-signal), it performs a courtship dance and transmits signal code-C2 again. The courtship dance will be continued until each courtship-signal is obstructed by other agent-robots. Figure 5. Piano Actuators 6. CONCLUSIONS We have presented a musical framework with interactions among plural physical autonomous devices via two different installations. The results of the second implementation attempting to generate music autonomously is as follows: Firstly, the implementation could obtain more impact through the utilization of a traditional and physical instrument. Secondly, the agent-robots generated and evolved music through their interactions and the feedback system. Thirdly, dynamic attributes existed, in which sounds were sometimes very few and sometimes plentiful. Finally, the attribute occasionally caused out of control as a sound installation. However, it can generate fluctuation which is very musical, according to circumstances. We will develop a method of control via each agent's behavior. In addition, there are many future works. First, We have to search for a battery which has a longer operating life. In the present version, the duration of activity for agentrobots is only 30 minutes. Second, there is the possibility of behavior evolution for agent-robots with GA or GP. It can realize advanced self-organizing of music in this framework. Finally, it is third implementation task with the framework for more practical purposes such as musical-controller or selforganizing instruments with swarming components. 7. REFERENCES [1] Ajay Kapur. A HISTORY OF ROBOTIC MUSICAL INSTRUMENTS, in Proc. ICMC07, September.2005, pp. 21-28. [2] Erol Sahin. Swarm Robotics: From Sources of Inspiration to Domains of Application, Swarm Robotics SAB 2004 International Workshop (July 17, 2004), 10-20. [3] Stuart Kauffman. At Home in the Universe: The Search for Laws of Self-Organization and Complexity. New York: Oxford University Press, 1997. [4] Maturana, H.R. and Varela, F.J. Autopoiesis and Cognition: the Ralization of the Living. Boston:D. Reidel Publishing Co., Dordrecht, Holland, 1980 5.3.2. Piano Actuator A physical instrument (piano) was utilized to generate sounds in this system. It builds up non-linear attributes and impact for the installation. Eight actuators are utilized to play the piano (see Fig.5). To control them, an I-CubeX was utilized. All actuators are designed to push each specific keyboard with its arm according to the control signal. The system sends the control signal to the actuators based on LEDs flashing on the agent-robots. In consequence, the piano is played. The system generates melody and rhythm. 121