Page  00000449 MusiCloth: A Design Methodology for the Development of a Performance Interface Lonny L. Chu CCRMA, Department of Music Stanford University lonny Abstract There is great interest in computer music in learning how to provide musicians, both composers and performers, with better interfaces to computing capabilities. However, since musical success is an aesthetic and cannot be determined from quantitative measurements, the needs of musicians are significantly different from the needs of many other computer users. Therefore, a distinct design methodology must exist for creating musical interfaces. This paper describes work on the design processes centered on the development of a performance environment called MusiCloth. Introduction One of the major concerns in computing today focuses on issues that reside beyond the technology of the hardware. It is becoming more and more important that the relationship between the user and the electronics is healthy and beneficial for productivity. As a culture, we are moving farther away from arcane command-line interfaces toward computing environments that leverage natural human abilities to provide the most effective interfaces possible. Mitchell Kapor has urged interface designers to be more sympathetic toward aesthetic and artistic endeavors, to "[design] for the full range of human experience" [Kapor, 1996]. The MusiCloth project is an extension of this idea within the context of computer music performance. The goal is to develop a design methodology for the creation of an immersive, real-time performance environment by making the computer as accessible and as controllable to a skilled performer as an acoustic instrument. Unlike traditional interfaces, however, the criteria for successful music production rely on aesthetic judgment rather than on quantifiable measurements such as efficiency. Therefore, although many design techniques may be borrowed from the fields of product design and interface design, special considerations must be made when designing for music. Design Components The overall design process for MusiCloth resembles the classic design cycle of need-finding, brainstorming, prototyping, and user testing. With these techniques, MusiCloth will hopefully evolve into an instrument that fulfills the needs of a performer while providing a user experience that is musically rich and fulfilling. From the above techniques, MusiCloth aims to satisfy four major concepts: 1. MusiCloth must be based on a robust conceptual model that supports music performance. 2. MusiCloth will be a multi-modal environment, using relevant visual, audio, and physical interfaces. 3. The interface of MusiCloth must rely on direct manipulation of virtual objects by the performer. 4. The environment must allow for the flexibility and power of control necessary for performing compelling music in real-time. 1. Conceptual Model The conceptual model is one of the most important factors in determining how successful an interface can be [Liddle, 1996]. If the model is not clear in the user's mind, the interface feels unnatural and forces unwanted complexities into the user experience. Therefore, it is very important to provide MusiCloth with a clear, yet aesthetically powerful,, conceptual model. The model on which MusiCloth is based is that of a hanging tapestry or curtain that produces sound when touched. Imagine walking up to a brightly colored tapestry that one can touch gently or vigorously, that has a variety of resiliency depending on how and where it is touched, and that possesses different textures along its surface. As one presses or strokes the tapestry, sounds emanate that are controlled by the gestural manipulation of the cloth. This model should supply an excellent relationship for a performer in creating music. Intimacy is a critical component in music performance and it is very important that the performer has a close relationship with the instrument. The tapestry model works because cloth is a very intimate material for people. It is an everyday item that our ICMC Proceedings 1999 - 449 -

Page  00000450 skin is always in contact with so it is an ecologically powerful material. In our society, we have an entire lifetime of experience dealing with clothing of different textures, weights, durability, and thickness. Add to this the many objects around us made of cloth such as curtains, couches, towels, sheets, and so on. This provides a tremendous knowledge base of what cloth is and how we should interact with it. MusiCloth will take advantage of this extensive, and personal, body of lifetime experience for the creation of music. Although intimacy with cloth and intimacy with music are quite different, there may very well be psychological and emotional consistencies in the way we deal with a variety of intimate experiences. By using this conceptual model, the design of MusiCloth may help answer questions such as this. 2. Multi-Modal Environment Part of the power of intimacy stems from the way we use multiple senses to gather various modes of information in the overall processing of an experience. For example, eating can be a very powerful experience. However, eating makes use of more than just our sense of taste. We smell the food, see the food, and feel the physicality of the food in our mouths. Likewise, in the case of music performance, we do more than simply hear the sounds coming from the instrument. With acoustic instruments, we can see the instrument during usage, and perhaps more importantly, we sense the way it feels as our hands or mouths manipulate the instrument's various mechanisms. These tend to be very powerful channels of feedback which are often treated as secondary characteristics. As Stanney explains, sensory redundancy can be a very powerful tool in making human interaction more vivid [Stanney, 1995]. For VR applications such as data visualization, vividness means making more information accessible more clearly. In the case of music, however, vividness is a rather nebulous concept. Music itself is so vaguely defined that it is difficult to understand how one can make it more "musical". It is much easier to hear music and make this judgement than to construct it with this goal in mind. Still, it is reasonable to believe that by carefully incorporating more sensory modalities into an interface, the process of making music with a computer can become a richer and more fulfilling process. The MusiCloth design addresses this issue by utilizing three primary sensory modalities: hearing, sight, and touch. Being a musical instrument, the aural feedback is the musical output produced in the environment. Currently, the sounds are created by an external MIDI device receiving instructions from MusiCloth. MusiCloth's visual feedback appears in the form of a computer-generated graphical display. This display is meant to represent the tapestry that the performer manipulates and should appear on a large screen. The display is implemented through a generic framework that exists as a polygonal mesh. Mesh View This mesh can be pushed and pulled and stroked and will adapt to these actions so that the visual representation reflects the physical actions that are imposed on it. The user can then design a graphical overlay to place on top of the mesh so that the public appearance of the tapestry is aesthetically pleasing and m e importantly, is related to the music being performed. Overlay View -450 - ICMC Proceedings 1999

Page  00000451 For example, in the above Overlay View, the graphic acts as a map telling the performer that certain events may happen when specific regions are manipulated. When a spot is pulled, the graphic changes shape to reflect this action. Simultaneously, this may trigger sound events that are unique to that particular spot as well as the specific gesture. Conceptually, the graphic should be large enough to seem like a real tapestry. Initial prototyping and research for this project is taking place on conventional computer monitors, but hopefully, the final version will appear on a large-scale display. By using displays that are several feet in dimension, MusiCloth can become a physically attractive interface that can leverage large, sweeping performance gestures as well as small, detailed actions. The large display also adds to the performance by providing a suitable means of visual stimulus for the audience during the course of the performance. The third feedback modality of the MusiCloth design is touch. Although the exact implementation of this feature is still undetermined, touch will be a critical component of the project since the model focuses on the intimacy and close pTiysical relationship between the performer and the instrument. Ideally, a haptic interface would be used to add this feedback channel to the design. Since the interface would work in cooperation with a large-scale display, the haptic device must allow the performer a great deal of freedom in movement, be able to accurately and precisely sense large as well as small gestures, and provide a broad range of haptic feedback. Currently, no such device exists that can provide all of these features within the project's financial budget. Therefore, the controller aspects of MusiCloth will be limited to technology that is available. The most likely scenarios are the use of a 6 degrees of freedom hand held controller or the use of glove devices that are capable of measuring finger and hand movement and perhaps supply limited forcefeedback capabilities. 3.Direct Manipulation Another key concept of GUls today is that of direct manipulation. This describes the process in which the user manipulates computerized object representations directly without having to go through an additional layer of abstraction. A common example is the use of windows and icons in contemporary GUIs. Instead of working wich files through text on a command line, the user directly manipulates icon objects that are strong graphical representations of files. The purpose of this technique is to provide the user with a more natural mapping of command to action. The incorporation of direct manipulation becomes even more important in the case of real-time music applications. Since so much of music is about feeling, a performer should not have to think about how to control the interface during the performance. Hopefully, the perception of direct access to the MusiCloth display will allow the performer to concentrate fully on the music rather than on the interface. In MusiCloth, direct manipulation is implemented on the surface of the graphical tapestry. The first stages of this project. use a traditional pointing device, such as a mouse, to access the tapestry on the screen. However, the goal is eventually to use hand controllers that allow more simple gestures to control the instrument. With a glove device, natural motions such as pulling, pinching, poking, or sliding become the means for manipulating the tapestry. This will provide an extremely powerful and effective sense of immediate control, thus allowing the performer to interact in a naturally physical manner with the interface. 4. Flexibility and Control One of the more intriguing aspects of computer music is that computers can be programmed to do what we want them to do. Unlike acoustic instruments, whose output is significantly determined by their physical attributes, the generation of a performance through a computer depends on programmed instructions. This can be a powerful tool for creative endeavors such as musical expression. Oppenheim elaborates on this issue through his work on "slappability" and "transmogrification" [Oppenheim, 1993]. Essentially, these ideas involve the formation of ideas in one form and then applying them to musical information in another form. In the software packages DMIX and Music Sketcher. one is able to use various modes such as graphical objects, text-based instruction sets, or note-based musical representation to formulate and store ideas. Then, using the processes of "slappability" and "transmogrification", one can apply the newly generated objects onto other objects. This capability allows for a system that is very flexible in usage as it adapts to the users' strengths. MusiCloth intends to reflect some of this power in its design. Although the goal is not to create a system as flexible as Music Sketcher, it is still important that MusiCloth provides the capabilities for a composer or performer to configure the environment to what is needed for a specific performance. This will be accomplished by allowing the environment designer to apply ICMC Proceedings 1999 - 451 -

Page  00000452 graphical, gestural, and sound map overlays onto the basic mesh framework. As seen in the above images, a graphical overlay can be freely designed to accomplish two purposes. First, it should provide visual cues to the performer as guidance for where and what to manipulate on the tapestry to produce certain sound events. Second, it should be aesthetically interesting in the visual domain for both the performer and for the audience to enjoy during the performance. At the moment, the graphical overlays exist as simple graphic files. This means that they can be created using common drawing or painting applications for intricate visual presentations. Similar in concept is a gestural overlay that acts in conjunction with the visual and sound maps. This map contains information relating to the gestural information provided by the performer and allows the designer to create overlays that instruct MusiCloth how to react visually and aurally to the performer's gestures. As MusiCloth evolves, these overlays will develop such that the function of each draws upon the information contained in the other overlays. This relates to the concept of "slappability" because it would allow the designer to use one mode of creation to influence the behavior of other modes. By implementing this feature, MusiCloth will become an extremely flexible and powerful tool for music creation and performance. Conclusion Although far from completion, the work done on MusiCloth thus far has provided useful insight on what it means to design an interface for music. Because music relies so heavily on the user's interaction and reaction to the instrument, an effective interface must provide the means for rich, yet intuitive, control. This has meant extending concepts taken from classic interface design. For example, the use of multi-modal feedback becomes extremely beneficial in providing information of sight, sound, and touch to which the performer may react quickly. Similarly, the idea of direct manipulation extends beyond using a mouse to click icons to a situation in which the performer is encouraged to use his or her own hands to interact directly with a large visual tapestry. As the MusiCloth project continues, many iterations of need-finding, brain storming, prototyping, and user testing will occur. Each cycle will bring a better understanding of the needs required by performance interfaces as well as a clearer view of a solid methodology for designing future music interfaces. [Oppenheim, 1993] Oppenheim, D. "Slappability: A New Metaphor for Human Computer Interfaction". ICMC Proceedings, Tokyo, 1993. [Oppenheim, 1996] Oppenheim. D. "DMIX: A Multifaceted Environment for Composing and Performing Computer Music". Computers & Mathematics with Applications, July, 1996, Vol. 32 (1), pp. 117-135. [Roth, 1996] Roth, R. "Music and Animation Tool Kit: Modules for Computer Multimedia Composition". Computers & Mathematics with Applications, July, 1996, Vol. 32 (1), pp.137-144. [Sorensen, 1985] Sorensen, V. "Visualizing Music with Computer Graphics". Proceedings of the 6rh Annual Conference and Exposition of the National Computer Graphics Association. 1985, pp.807-815. (Stanney, 1995] Stanney. K. "Realizing the Full Potential of Virtual Reality: Human Factors Issues That Could Stand in the Way". Proceeding of the IEEE 1995 Virtual Reality Annual International Symposium, 1995, pp.28-34. References [Borchers & Muihlhauser, 1997] Borchers, J. & Miihlhauser, M. "Musical Design Patterns: An Example of a Human-Centered Model of Interactive Multimedia". Proceedings of the IEEE International Conference on Multimedia Computing and Systems, 1997, pp.63-70. [Fels, Nishimoto, & Mase, 1997] Fels, S., Nishimoto, K., & Mase, K. "MusiKalscope: A Graphical Musical Instrument". Proceedings of the IEEE International Conference on Multimedia Computing and Systems, 1997, pp.55-62. [Kapor, 1996] Kapor, M. "A Software Design Manifesto". Bringing Design to Software, ed. Terry Winograd, ACM Press, 1996. [Kobayashi & Schmandt, 1997] Kobayashi, M. & Schmandt, C. "Dynamic Soundscape: mapping time to space for audio browsing" Human Factors in Computing Systems CHI 97 Conference Proceedings, 1997, pp. 194-201. [Liddle, 1996] Liddle, D. "Design of the Conceptual Model". Bringing Design to Software, ed. Terry Winograd, ACM Press, 1996. [Norman. 1988] Norman, D. The Design of Everyday Things, Doubleday, 1988. - 452 - ICMC Proceedings 1999