Page  00000001 LITEFOOT1 - A FLOOR SPACE FOR RECORDING DANCE AND CONTROLLING MEDIA Niall Griffith Mikael Fernstr6m Centre for Computational Musicology & Computer Interaction Design Centre, Music, Dept. of Computer Science & Information Systems, Dept. of Computer Science & Information Systems, University of Limerick, Ireland. University of Limerick, Ireland. Email: Email: Tel. +353 61 202758 Tel. +353 61 202606 Abstract This paper reports the design and implementation of a proximity sensor based floor that is capable of tracking a dancer's steps, transmitting these to a linked computer for analysis and/or to drive other media. The design rational is outlined in the light of previous dance floors that have been developed to control performance. The aim is to develop the floorspace for recording and representing dance: for use in dance research programmes, choreographic experimentation, performance and musical composition. Keywords: Interactive Composition, Movement Sensing, Musical Interfaces, Light sensors, Dance Recording. 1. Introduction Although dance is an almost universal human recreation and dance performance is likewise culturally ubiquitous, the idea that a dancer's or performer's feet can be used to play a musical instrument in the way that, for example, a pianist's hands are used to play the piano, is relatively new, and largely unexplored. However, the tapping of a dancer's steps can be an important rhythmic component as well as a controlling element in a dance performance. It is the extension or transduction of the natural percussive sound of dance steps into synthetically generated sounds, or their use to control other media - visual arrays, perhaps even an olfactory or tactile stimulus that is new. The last five or so years has seen a number of prototype systems that have used sensors embedded in or comprising a floor surface to capture footsteps. The work of Johnstone, Pinkstone, and Paradiso (Johnstone 1991, Pinkstone 1995, Paradiso 1997) all describe systems that use the contact, force or weight of a performer to capture footsteps and subsequently to generate sonic events. The design reported here uses proximity sensors to precisely locate a dancer's foot contact with the floor. The reason for making this choice may not at first appear to be obvious, However, the hardware Credits: The LiteFoot Project is a collaboration between The Centre for Computational Musicology and Computer Music and The Interaction Design Centre at the University of Limerick. The development of the floor involved the authors and Liam Bannon with user input from Catherine Foley. The authors designed the floor, its software architecture and functionality. Mikael Femstrim designed the sensor system. The software was coded by Padraig Finnerty and the floor was fabricated by Joe Murray and Mona Fernstrim. More details on the project can be found at

Page  00000002 and software design emerged from a precise user requirement, namely the wish of researchers in the Irish World Music Centre in the University of Limerick to be able to record and analyse traditional Irish dance forms. Trying to achieve this end meant investigating several commercially available systems for tracking dance in 3D. However, it soon became clear that these systems either involve interference with a dancer's movement and/or are very expensive. Secondly, we investigated the ideas involved in a number of floor sensor systems. While the floors reported in Johnstone, Pinkstone, and Paradiso et~al. are excellent performance systems, some of their characteristics make them unsuitable for the end we were working towards. The following section reviews these systems not with the intention of criticising them but to present the perspective that has led to our own design. 2. Previous Floor designs The PodoBoard (Johnstone 1991) was developed by Eric Johnstone at McGill University to extend the use of clackage, a form of seated dance that takes place (traditionally) on a small wooden floor. The PodoBoard provides an accurate set of co-ordinates of contacts with the floor, which is a matrix of 1 inch square aluminium tiles. The floor's reaction to footsteps is extremely fast, probably as fast as 10 nanoseconds. However, although very impressive in performance terms this system has several drawbacks. The system works through the shoes completing an electrical circuit - these have metal contacts at toes and heel. This means firstly, that the dancers shoes have to be specially adapted before a dancer can make use of the floor. Secondly, the adaptation made might not be either particularly easy, and the result may not facilitate performance where, for example, soft shoes are being used, or for that matter be very safe where free-moving dance proceeds rapidly over a smooth surface. Thirdly, the system seems to be critically dependent upon good electrical contact being made between shoe and floor. One solution to the specialised adaptation of the heel and toe of the shoe might be to use electrically conductive rubber. This would be more acceptable in dance, helping the dancer grip the floor, and it would not significantly slow down tracking - reaction would probably still be within 1 millisecond. However, it would probably exacerbate the electrical contact problem. Electrical circuits are notoriously susceptible to bad contact, and much dance involves sweeping and momentary contact with the floor. The PodoBoard seems likely to miss the light foot contacts involved in some delicate dance steps. A rather different solution to using footsteps was developed by Russel Pinkstone at the University of Texas (Pinkstone 1995). This floor uses force sensors arranged in strips. The granularity of the system is about 6 inches. This is again a very useful floor, but the sensor technology is relatively expensive. Apart from this the floor has two characteristics that make it less attractive as a potential medium for recording dance steps. Firstly, the system is not designed to deliver the precise co-ordinates of the footsteps that are made. It might be possible, by using orthogonal, overlapping strips to calculate the position of an applied force, but this seems to very difficult, and as the floor size increases it is likely to be impractical. Secondly the floor involves an analogue to digital conversion that is relatively slow - approximately 2 milliseconds per reading. This translation effectively rules it out as a viable method of tracking dance that can often achieve step rates of as fast as 30 steps a second (approx. 1 per 33.3 milliseconds). MIT Media Lab's M~agic Carpet (Paradiso et~al. 1997) also uses the force of a person's footsteps. The technology used is based on cable insulated with piezoelectric material that responds to compression and bending by producing a change in capacitance. The output is multiplexed and the wires are scanned 60 times per second. The floor is very sensitive to foot pressure. The system was designed to allow people in public spaces to create and manipulate a sound environment. There are a number of aspects that again make it less useful as a medium for recording dance. Firstly, the floor uses a grid of wires occurring every 100 mm (4" approx.). This is a relatively coarse grained matrix when considering the translation of steps into a choreographic representation, such as LabaLnotation (Hutchinson 1974). Also, the M~agic Carpet responds

Page  00000003 relatively slowly. Its temporal resolution is approximately 0.5 of a second. Although the floor is scanned at a higher rate the effective response rate is reduced because the wires are quite noisy and low pass filter on the electrical inputs limits the bandwidth of the system. (This is probably increased by the Doppler radar sensors at the side of the mat). Again this rate of detection is unable to cope - for example - with Irish traditional dance where step rate can rise to the 30 per second already mentioned. Also, it means that the resolution of the floor cannot be increased easily. If the grid was finer the wires would be closer, the noise levels would increase and the response time of the floor would be further reduced. Thirdly, and most significantly for tracking dance steps, the Magic Carpet cannot guarantee to locate the steps of more than one foot at once. This is due to the use of crossed wires to locate the co-ordinates of each foot. This means that one foot can shadow another because the system scans the periphery of a foot. A footstep that is larger blocks or overlaps another step that falls along the same x or y co-ordinates, so that smaller footfalls are invisible within the boundaries of the larger. So while it is possible to know that there are two objects on the mat it is not always possible to determine accurately where the objects are. This is a characteristic that may be worked around in the context of performance. It is a disadvantage if the floor has to be capable of transmitting precise co-ordinate information. 3. Specifying the floor A number of considerations seem to be of prime importance in the design of a floor space that is capable of recording footsteps as well as being used as an input device for generating and controlling creative processes. In fact while some of these requirements can be compromised in a creative system, and become in effect part of the definition of the 'instruments' capacity, these same characteristics can render a floor effectively unable to track steps. We wanted to use the floor to provide an accurate record of a dancer's steps. Any increase in the device's specification that arise from it being required to track steps also refines its definition as an interface and musical input device, i.e its specification as a musical instrument. This can only be an advantage. The principle requirements for a dance floor being able to track dance steps are: * The floor and any associated system should be able to respond to and record all the steps made. * The floor should be able to record the exact position of the dancers feet. * The floor should be able to discriminate multiple feet and multiple dancers. 4. LiteFoot Floor Design The prototype LifeFoot floor is a 1.76 meter square, 10 centimetres high slab, filled with a matrix of optical proximity sensors. When a person stands on the floor the location of their points of contact with the floor are detected by the sensors. An embedded controller linked to the floor scans through the sensors every 10 mins. It establishes for each sensor position or 'pixel' of the floor whether or not the state of that pixel has changed or not since the last scan. It then transmits any detected change to a PC. The physical construction of the LiteFoot floor consist of four layers: * A structural, honeycomb wooden base. * A layer of printed circuit boards carrying the proximity sensors and associated electronic components. * A layer of plywood recessed with holes to accommodate the sensors. * A protective upper layer of polycarbonate that sits over the sensors. The sensors are arranged in blocks of 16 that are assembled on printed circuit boards (PCB's). There are 121 PCB's altogether, each is 160 mm square and the resolution of sensors across the floor is

Page  00000004 40 mm. The current resolution of the floor was motivated by the need to prototype and prove the viability of the floor's design. The only physical limitation on the ultimate resolution of the floor is the size of the sensors, a few mm2 in vertical cross-section. Figure 1 The arrangement of the sensors on the surface of the dance floor. The sensors are hidden beneath the floor's upper surface and consist of a pixel array of 44 x 44 (1,936) sensors cover a 1.76 m2 area at a density of one sensor per 40 mm. At present the floor can be used in two receptive modes. The difference is not physical, but lies in the operation of the embedded controller software. The first mode is a reflective mode. Here the footsteps are detected by the proximity of an object causing a reflection of light back to the sensor that emitted it. This is most effective when the dancer wears shoe that have light reflective soles. The second mode is a shadow mode where the floor is flooded with light and the footsteps stop that light from entering the sensors. The two modes are effectively positive and negative. The floor is equally responsive in either mode. 5. TipTapToe and FootWare: Software Modules and Libraries Apart from the physical floor, LiteFoot consist of three software libraries. These correspond to two layers of control on the PC and the embedded controller software. Figure 2 shows the overall arrangement of the software modules in LiteFoot. The three main software components comprise a set of low level routines on board the controller, TipTapToe which comprises a core library of C++ classes to allow data to be read from the floor and written to a common data area and external file. The third set of software is FootWare. This is a set of software drivers to allow the configuration and use of audio media such as MIDI, visual display etc. The fundamental principles underlying the LiteFoot software are that the software should enable simultaneous modes of independent use, i.e. performance/composition and recording/analysis to take place concurrently. The users - whether they are composers, dancers, choreographers or dance analysts - are regarded as the ultimate guide for the systems development. For this reason we have adopted a modular approach that will allow the easy extension of the set of software drivers. Like the hardware the software has been motivated by the need to use the captured dance steps as a real time control medium in performance and as a means of recording the dance, including that multiple foot steps should be recorded in real time. On the PC, the events can be processed in a number of ways: Thus, far our concern has been to verify the basic hardware design and establish a set of basic software that will facilitate a number of independent software processes (potentially distributed over a network of machines) to have access to a

Page  00000005 shared memory cache and file of floor events. We have also implemented two basic drivers for the control of visual and audio effects. The first of these drivers is a matrix display of the state of the floor, that is to say what sensors are currently 'on', coupled with a time delayed decay function that decreases the intensity of the displayed pixel. Thus a trace of footsteps is displayed that dissolves as time passes. The second driver is a simple MIDI device that maps for a chosen instrument the floor positions to a pentatonic2 scale. The floor can also divided up between instrumental sounds. These two drivers while being very simple are nevertheless effective mediators of performance, if somewhat inflexible. The floor has already been used in a large public concert for a short traditional Irish dance, and a free form modern dance. Dance Floor PC Media 1 FootWare IMedia 2 SZMedia N SData cache I, t,71 l lI l lI Figure 2. Overall organisation of the hardware and software in LiteFoot. Currently a fully specified MIDI mapper is being developed and a Labanotation program that will convert the detected dance steps into this symbolic notation widely used in choreography and dance recording. This will allow the data gathered by the floor to be input to a Labanotation editor for further manipulation (Hunt 1989). 6. LiteFoot and Performance The LiteFoot floor has had several public outings as a performance medium, including one in a major public concert. In this concert the floor was attached to a PC running the two drivers outlined above. (The first maps the data to a pixellated visual array, with a fade out. The second maps the floor to a Pentatonic Scale.) Although, both of these drivers were extremely simple the combined effect in the context of the performance, of both traditional Irish dance and improvised modern dance, was perceptually engaging. Viewing a video of the performance some time after the event we were struck by the potential of the device, even though the routine itself had been choreographed literally in a few minutes before the performance as all rehearsal time had vanished in removing last minute hardware and software glitches! The floor itself is very sensitive to subtle inflections of foot movement and these were reflected both visually 2 The pentatonic scale was chosen for the first performance as appropriate for Irish traditional dance.

Page  00000006 and aurally. The dancers steps became the music with the rhythm and tempo of the dance augmented by being transduced into the additional dimensions of pitch and timbre. The success of the performance confirmed to us the potential of the floor. 7. Future Development Our assessment of the usefulness of the LiteFoot dance floor is only just starting. Apart from its potential as a performance medium, the concert also indicated that it could be used as a compositional aid. One initial approach to realising this aspect might be to follow the lead of Pinkstone and Johnstone (Johnstone 1991, Pinkstone 1995) who used the higher level musical manipulation languages MAX and Soundscape to control the compositional process. More recent developments include extending the TipTapToe software so that a larger floor space, perhaps linked to a number of PC's can be accessed and used as a single floor. Secondly we are developing both a fully specified 'mapper' that will allow the user to control on-line the mappings between various aspects of the floor and media events, e.g. MIDI, extending the simple mapping of the x and y co-ordinates of the floor to MIDI events. Other dimensions and possibilities will involve using the foot force, and step speed to drive events. Also, at present the floor is mapped as a single space, so it will be useful to explore both being able to edit subdivisions of the floor in real time, changing the mappings within different areas, as well as differentiating the granularity and dimensionality of mappings. These are aspects that we will explore more fully as the system develops. There are also other educational and therapeutic uses that seem to be possibilities. These include using LiteFoot as: * An auditory/musical play space: Where an individual has limited co-ordination or motor capacity or where a child has difficulty manipulating a complex interface - (and many musical instruments do involve complex interfaces!) - it may be possible to use a sensitive dance floor like LiteFoot to facilitate the creation of auditory and visual or other media streams. Rhythmically rolling on the floor's surface, or swishing the arms or legs (an electronic version of snow angels perhaps?) is enough to create, albeit simple, nevertheless structured sequences in whatever medium is being driven3. The stimulation given to the 'composer' by such a simple 'interface' is very difficult to assess, but as the created events are perceivably (causally) connected to body movement they may offer some people a unique situation in which they to can enter a creative loop that is denied to them in other situations where more complex interaction is needed. Certainly, this kind of system would be similar to SoundBeam (Ellis, 1997), that has demonstrable benefits, but using floor contact rather than movement in the air. * An arena for teaching dance: It is feasible to develop drivers that having recorded the steps of a dance - perhaps danced by the teacher - would then be able to compare this with the students steps and provide feedback on how the students performance should be changed. * To generate adaptive dance accompaniment for practice. This would allow the student to be accompanied by music played at the right tempo for a particular skill level, and to develop a performance. 3 In one public event children played on the floor, and although the floor was overloaded with 10+ children 'on-board', several of the children 'latched-on' very quickly to the rhythmic possibilities of the space.

Page  00000007 8. Summary In this paper we have presented the current state of our development of a proximity sensor dance floor. The floor responds to footsteps on its surface, and these are recorded via an embedded controller that scans the state of the floor and a PC that maintains a shared data area and also saves a record of the dance to disk. The goals of this project are to use the information gathered by the floor in various ways: * Creating dance performance. * To extend the use of dance steps as an element in a compositional process. * To make a dynamic representation of dance, using a movement notation system (Labanotation). The potential of the floor as a musical input and performance device lies in the fact that it can be used in a relatively large scale situation, and because it captures accurately 2D co-ordinates it can be used to calculate a wealth of movement related measurements. The measurements that we have experimented with so far to drive media are a small fraction of those open to capture with an expressive performance on the floor. The comments of dancers who have experimented with the floor suggest that the definition and direct control of a more plastic and realistic (expressive) set of mappings from the floor readings is needed to realise this potential. This is our current objective. Acknowledgements We would like to thank Kevin Ryan for his financial support and the Department of CSIS for financial support and the loan of equipment during the floor's development. Thanks also to Micheil 6 Stiilleabhain for giving us the creative space in which to develop and premier the floor. References Camurri, A. (1995) Interactive dance/music systems. Proceedings of the 1995 International Computer Music Conference (pp. 245-252). San Francisco: International Computer Music Association. Ellis, P. (1997) The Music of Sound: a new approach for children with severe and profound and multiple learning difficulties'. British Journal of Music Education, 14:2, pp. 173-186. Hunt, F.E.S., Politis, G. and. Herbison-Evans, D. (1989) An Interactive Graphical Editor for Labanotation. Technical Report 343, Basser Department of Computer Science, University of Sydney. Hutchinson, A. (1974) Labanotation. 2nd Edition. Oxford University Press. Johnstone, E. (1991) A MIDI foot controller - The PodoBoard. Proceedings of the 1991 International Computer Music Conference (pp. 123-126). San Francisco: International Computer Music Association. Paradiso, J., Abler, C., Hsiao, K., Reynolds, M. (1997) The Magic Carpet: Physical Sensing for Immersive Environments. Proceedings of the International Conference on Computer Human Interfaces ( pp. 277-278). Association of Computer Machinery. Pinkston, R., Kerkhoff, J., and McQuilken, M. (1995) A Touch sensitive Dance Floor/MIDI Controller. Proceedings of the 1995 International Computer Music Conference (pp. 224-225). San Francisco: International Computer Music Association.