Page  00000001 A survey of agents based real-time interactive systems Peter Beyls Abstract The present paper discusses four models of spatially distributed agents engaged in social interaction. We explore emergent functionality in a complex dynamical system as an alternative to explicit design in the realm of real-time interactive systems. This work is heavily inspired on computational paradigms from the discipline of artificial life. 1. Introduction Software agents technology is booming and has now also received wide spread attention in the arts. In particular, models of social computing inspired by collective behaviour in biological workspaces are extremely relevant. For instance, the emergent functionality as witnessed in colonies of interacting insects can make for an attractive approach in solving problems of musical interaction. The designer of real-time conversational musical systems cannot depend on knowledge-based approaches since the quality of the interaction would be restricted to situations anticipated by the designer. In addition, it is very difficult to model the dynamics of interaction in explicit rules. One is forced to adopt selforganisation as an alternative to explicit design since we wish to remain adaptive to an unpredictable environment e.g. man and machine interacting in free improvisation. Artificial life studies abstractions of life inspired (though not necessarily) on forms of biological life and aims its understanding in terms of information dynamics and behavioural complexity. Agents are the basic building blocks which engage in collective decision making. For the purpose of our work, an agent is defined as a simple virtual organism which behaves according to the nature of its personal character as well as according to the forces of a simulated social climate expressed in external conditions and inter-agent affinities. We have implemented various models mostly based on the principle of conflict resolution. Coordinated global musical behaviour issues from local interaction only. Agents function as a society of interacting virtual musicians that possibly also listen to an external human musician. Agents wish to express their personal character while, at the same time, they wish for integration into a larger social whole. So the tension between integration and expressive forces makes the system behave in a non-linear fashion. The present paper reports on agents moving in continuous two-dimensional space positioning themselves according to mutual affinities and providing real-time visual feedback to the user. Complex locomotion emerges from local interactions (Ventrella, 1994). Other implementations are based on cellular automata (CA) in which agents move in a twodimensional grid. The behaviour of the CA follows from a lookup-table rule. This rule is edited in real-time according to a critical analysis of incoming midi streams. In addition, the rule will modify itself in order to suggest a salient musical climate in case external input is missing. Again, one has created a system with a personal, internal drive that remains responsive and adaptive to disturbances by external input. Some programs are implemented in HMSL, others in LISP. All share the same principles of social computing: the user is seen as a participant (wetware agent) in a society of software agents. This makes for a robust and flexible system following the spirit of current research in the discipline of artificial life. Next paragraph describes four different implementations. Example 1 and 3 accept input from an external human performer while the

Page  00000002 other examples function as autonomous, selfregulating micro-worlds. For every example, we provide a single snapshot which gives a visual impression of the unfolding activity. 2. Implementation Example #1....................-........:................. e - - - - - -: - - -:.......... I ^..-JI: sensitivity expressed as a critical distance) and, finally, a tendency to listen to the (wetware) agent at the other side of the screen, i.e. the human performer. Incoming midi events may result in new agents being born which immediately start interacting with agents from previous generations -- this feature gives the system a unique type of long term memory. This approach allows the exploration of both autonomy in a self-regulating society of interacting virtual musicians, as well as having the agents act as a living interface towards the human performer. In both instances the initial physics (relationships between individual agents, overall energy etc.) are designed explicitly by the user, the final behaviour however follows from the complex interplay of many local forces, global structures emerge from self-organisation in a distributed collective. Example #2 This program is part of Oscar, a large modular system which simulates the psychology of a living musician (Beyls, 1988). The program attempts to execute a plan while simultaneously accommodating unpredictable external input from a solo human performer. It functions as a pattern directed inference system trying to blend its own goals with external pressures into a musical conversation which makes sense. In short, its functionality issues from a conflict resolution between integrative and expressive forces. Oscar is primarily a knowledge based system extended with a sub-symbolic module; a collection of agents moving in 2d space, first extensively documented in (Beyls, 1990). Agents are midi player objects which dissipate energy and behave according to the physics of the society they live in. Agents express affinities towards each other, they try to position themselves according to preferential distances. The overall goal is to minimise the stress in the society by adjusting their position in 2d space -- since all agents are interconnected the system exhibits all sorts of temporary point and cycle attractors. The melodic output of every agent depends on 3 parameters: the agents inclination to express its own character in isolation, its willingness to listen to nearby fellow agents (as function of a, * ~ +++* A S** ** ** * J This program introduces a slightly different type of conflicting requirements driving the system dynamics. A fixed number of agents, playing on a private midi channel, is configured in a circular network. Every agent is driven by two forces: a desire to play in harmony with its local context i.e. nearby agents, and second, a desire to develop themes in isolation. It is important to note that there is a finite, limited global energy supply. When integration supersedes expression, agents will search for other agents and when close enough will exchange musical material. A visiting agent also borrows energy from the other agent making it bigger, thus better seen by the society as a whole. The conflict resides in the fact that the agent

Page  00000003 becomes both stronger as well as less flexible (its step size decreases). On the other hand, a weak agent will produce less output but has a greater probability to hit another agent. This means that change will increase the possibility for further change: positive feedback occurs -- which is typical for any self-organising system. While engaged in effective interaction, agents also exchange intervallic material which is modified according to the current status and personality of the receiving agent. Consequently, the system sounds like a group of improvising musicians producing coherent results without the intervention of a supervising structure. Example #3 L... ~... S-.;.. ' A' R. P, - - *. 7 I~*, -..,. *. *A. j~ 1./?' _____^ since their values fade in time -- the decay parameter represents the external conditions of the ecosystem. A cell disappears when its value hits zero. The agent explores the representation by moving in 2d space producing a catalogue of all currently perceived events -- its left and right eyes having a parameter controlled but adaptive limited viewing distance. Now, the agent will move towards the area where most events are seen since its objective is to collect energy from these events in a which for survival. If no events are perceived, it will move in a random direction, when an equal number of events is seen left and right, it will move straight ahead. In any case, the agents new position will provide a new particular point of view of the scattered events i.e. potential energy resources. In addition, the critical viewing distance is adapted inversely proportional to the amount of events perceived. The agent holds two midi player objects (HMSL terminology) which continuously build melodies from the current perception of the evolving environment. The overall result is a robust performance system which accepts external stimuli and accommodates them in a long term memory structure which automatically keeps track of an unlimited input stream. Viewing the agent as being embedded in the environment, albeit in a simulation -- it is not grounded in any physical sense, provides a flexible, dynamic model for connecting the behaviour of human and machine performers within the realm of interactive composing. Example #4 The sound creatures program is a test bed for studying the behaviour of a large society of interacting sound creatures, software agents with proximity sensors and effectors affecting their position in 2d space. It resides on the hypothesis of behaviour-oriented decompo-sition and the exploration of emergent properties in a complex dynamical system. When instantiated, the agent inherits a large collection of methods -- a repertoire of simple behavioural modules which create links between sensing and effecting. This immediate functionality is particularly welcome in a real-time system -- having to reason over symbolic representations would indeed be far too time consuming. For instance, the collect_neighbours method will make a list of all creatures within view of the agents' critical Next example is inspired on the robot ecosystems approach which stresses a heavy emphasis on the interaction of hardware agents and their physical environment (Steels, 1993) as well as the behaviour-oriented approach suggested in (Brooks, 1991). Robots are autonomous entities equipped with sensors and effectors, they build their own representations and relate action patterns to unpredictable environmental conditions and limited energy resources. Our simulation actually implements the exploratory behaviour of a single software agent in a landscape of midi events. The agent has two sensors allowing for the detection of the position of midi objects in 2d space. Incoming midi events are mapped to 2d space using an interval transition network similar to a Markov lookup matrix -- an analogical representation is thus created. A matrix cell holds an energy value, updated according to the velocity and duration of midi input events. Cells have a limited life span

Page  00000004 distance as well as set its collision detection flag if needed. The mating_condition method returns true if two interacting agents are of the opposite sex. Instance variables include xy position, energy, status (awake or asleep), sex, age, internal clock holding the time delay since its last action, current angle i.e. direction in which it is moving, a history of its most recent trajectory in 2d space, variables which articulate its melodic output, angleincrement etc. There is no central control agency, all behaviours compete to be selected in confrontation with a given set of circumstances. The top priority goal of any creature is to engage in social interaction with other creatures. A creature will randomly explore the environment and adapt its sensitivities according to the current context e.g. if the agent is overwhelmed by too many stimuli it will decrease the value of its critical distance instance variable. Various things may happen while interaction takes place; the agent may invert the value of the angle in which it is moving or select a new, random angle when colliding, a melody is constructed according to the agents' energy level and current neighbourhood (akin to neighbourhood interpretation in non-discrete cellular automata). Sometimes, if two agents meet in isolation from the rest of the society, they may swap their angles making them bounce away. When many agents are in close proximity, a selection procedure singles out agents for immediate interaction introducing instability in the group as a whole thus making some agents available for future interaction. Unexpectedly rich behaviour emerges from a relatively small repertoire of interlocking actions. Certain interesting and co-ordinated global behaviour emerges without being programmed explicitly -- which explains the strength of the system. For instance, agents may engage in a digital dance, by moving in a parallel or twisted, wave-like pattern. Two agents trapped in a corner may be freed from isolation when visited by another agent. In short, social relationships like aversion and attachment emerge from a set of basically very simple actions given the right environmental conditions. The implementation offers real-time performance and takes full advantage of ODE; the Objectoriented Design Environment as part of HMSL (Burk, Polansky & Rosenboom, 1991). We are in the process of writing another version in the Java programming language as to make these virtual sound creatures live on the Internet in the near future. 3. References Brooks, R. 1991 Intelligence without reason In: Proceedings of the IJCAI, Los Angeles, Morgan Kaufmann Pub. 1991 Beyls, P. 1988 Introducing Oscar. In: Proceedings of the ICMC, Feedback Editions, Cologne 1988 Beyls, P. 1990 Sub-symbolic approaches to musical composition, A behavioural model. In: Proceedings of the ICMC, Glasgow 1990 Steels, L. 1993 Building agents out of autonomous behaviour systems In: Steels and Brooks (eds.) The artificial life route to artificial intelligence. Building situated embodied agents. Lawrence Erlbaum, Ass. New Haven Burk, Polansky & Rosenboom 1991, HMSL -- Hierarchical Music Specification Language, Frog Peak Music, Hanover, NH Ventrella, J. 1994 Explorations in the emergence of morphology and locomotion behaviour in animated figures, In: Artificial Life IV Proceedings, MIT Press 1994 4. Acknowledgements I would like to express my gratitude to Larry Polansky, Luc Steels, Gary Lee Nelson and Jeffrey Ventrella for discussions and productive feedback.