Page  00000001 Tangible Sound: Musical Instrument Using Fluid Media Tomoko Yonezawa tI and Kenji Mase t t ATR Media Integration & Communications Research Laboratories 2-2-2, Hikari-dai, Seika-cho, Soraku-gun, 619-0288 Kyoto, JAPAN $ Keio University Graduate School of Media and Governance 5322, Endo, Fujisawa-shi, 252-0816 Kanagawa, JAPAN Email: {yone, mase } @mic.atr.cojp ABSTRACT In this paper, we introduce" Tangible Sound' a musical instrument with a novel user interface that uses water. Like music, fluids cannot be physically grasped because their shape is constantly changing. We thus believe that water is a suitable interface for performing flowing music. We created a live, hands-on installation that uses the flow of water as an input medium to control the intuitively appealing feeling of musical flow. With our instrument, performers interact with water flowing from a faucet into a drain. We have developed a method for measuring the volume of the water flow and for generating music from this measurement. This installation leads to the novel concept of" Source and Draing' for programmable musical instruments. Finally, we consider the potential of this special interface for musical instruments and interactive arts. 1. INTRODUCTION Tangible Sound is a musical instrument with a novel user interface based on physical interaction with water. Water is a suitable interface for performing music because fluids and sounds have common characteristics. Most importantly, the shapes of both media change over time and thus cannot be physically grasped. Based on this innovative approach we created a live and hands-on installation also called Tangible Sound. Among the various fluids, water is most frequently used in daily life as an essential resource. Therefore, a musical instrument using water should be intuitively appealing. Although previous installations and interactive artworks have used water, they have not exploited its full potential to serve as a musical medium. Our installation thus aims to find practical applications of fluid media for musical instruments. The applications that we have in mind for Tangible Sound are essentially in entertainment. We can easily imagine that variations of the installation described in this paper could be integrated into our living environments, whether in kitchens, bathrooms, or public swimming pools. 2. BACKGROUND Classical compositions are generally limited by the use of a twelve-tone scale, and music is formed within this framework with reproductions of performances and expressions. Traditionally, composers have created pieces that are interpreted by musicians and listened to by an audience. Modern instruments and art installations, however, are offering a new perspective on musical experiences. The user can be at the same time the composer, the performer, and the auditor of an improvised musical flow. This observation has led us to the design of the Tangible Sound instrument, which makes the musical experience intuitive, tactile, and enjoyable. An important characteristic of sound and music is that they change over time. Like other fluids, water has the same characteristic. Furthermore, the tactile perception of water is similar to the perception of sound and music (Figure 1(a)). For people inexperienced in musical theory, it is not easy to understand the dynamic flow of music. Also, analyzing music by focusing only our auditory attention is difficult because the flow constantly changes and cannot be controlled. Considering these characteristics, we have applied water as a medium for musical performance. water shape musical flow time line (a) K- time musical flow (b) Figure 1(a): Common characteristics of temporal change Figure 1(b): Original concept of Tangible Sound Another inspiration for our system was the observation of several" instrumentý' that highlight the relationship between natural elements and music. Commonly found in Japan, wind chimes, shishi-odoshi', and sui-kin-kutsu2 produce sounds when affected by the flow of water and air (Figure 2). We thus decided to design an instrument based on users interacting with natural elements to produce sounds. Finally, it should be noted that water is used in many occasions in our daily life, whether in shower rooms, kitchens, pools, washing machines, or fountains. For that reason, we believe that the use of water as a medium might increase the opportunity for people to interact with musical instruments and to" feel' the music on a daily basis. We will demonstrate the live installation to provide participants with hands-on experiences. The concept of" Tangible Sound' will become clearer when people touch water and play music with the fluid media. 1 Traditionally, this instrument was activated by flowing brooks, and the sound was intended to scare deer, which dameged farmers' fields. Nowadays, this device is used as a garden decoration and mood-setter. 2 Sui-kin-kutsu is a natural sound instrument for Japanese gardens that uses a jar to produce reverb and resonance of the water dropping sound.

Page  00000002 3. RELATED WORK There are several examples of interactive systems that use water and shishi-odoshi use bamboo and water flow to produce sounds. Recently, many Multimedia Art and Interactive Sound Systems have integrated computers, sensors and other media. But in general, these installations do not consider water as a tangible input medium. Shishi-odoshi Sui-kin-kutsu Figure 2: Shishi-odoshi and sui-kin-kutsu Sugihara (1999) s Water Display uses water as an expression medium, but that system only exploits the surface of water as a projection screen. Ui and Setoh (1998J s Wave Rings is an interactive art installation where speakers are placed under water, which serves as a visual actuator. In other words, sounds produced by the speakers result in movements observable on the water surface. Here again, water is not used to get input from the users. In our system, we have used water as a tactile actuator that supports the generation of music by the physical manipulation of water. Ishii (1998) introduced the concept of Tangible Bits. Water and air are both considered ambient media, but no distinction is really made between them. Our perspective is slightly different: while both elements are not graspable, water is tangible and air is not. In our research, we have designed a musical instrument to take advantage of these properties. Perhaps the work most closely related to Tangible Sound is the installation by Sakonda (1998), where a photo TR sensor is used to convert waves on the surface of water into sound waves. Indeed, this installation suggests an analogy between the shape of water and the " shape" of music. Our work is closely related to all of these projects, but it also introduces original aspects. In particular, our installation highlights the direct and tactile interaction between the user and the water. 4. SYSTEM DESIGN 4.1. Concept In order to use water as an input medium, it is of course necessary to measure some of its properties. For instance, the quantity or the mere presence of water within a receptacle can be easily measured. Also, variations in its flow, its temperature or in the way light reflects on its surface can be monitored by various sensors. In Tangible Music, the flow of water between a faucet and a drain is measured to generate music (Figures 1(b), 3). We were motivated to use a faucet because it is a device with which people interact very often and thus very intuitively. In order to get input from the user, we have developed a method for measuring the volume of a water flow. As shown in Figure 3, the flow is measured both in the upper part of the installation (where water comes out of the faucet) and in its lower part (where water flows into the drain). Computing the difference between these two values provides a simple technique for monitoring the user's input indirectly and thus for generating sounds according to the movement of water. Faucet upper volume of water Sensor 1 User's input water flow I lower volume of water Sensor 2 Figure 3: Volumes of upper and lower flows The direct manipulation of water provides a very intuitive interface for a musical instrument. This configuration has led to the novel concept of" Source and Drains," which is applicable to traditional wind instruments. Indeed, with such instruments, performers generate a source of a fluid (air) by breathing into a mouthpiece. By interacting with several holes or switches, they then specify drains for this fluid (Table 1). The analogy with wind instruments had important implications for the design of Tangible Sound. We realized that the performer should be able to create different tones by specifying different drains for the flow of water. Our installation thus integrates four different drains. Table 1: Comparison between Tangible Sound and Traditional Wind Instruments 4.2. Hardware System Configuration. upper tank Pu''''' SOURCE User's Input Main DRAIN S and Sub-DRAIN lower tank (b) (a) Figure 4(a): View of Tangible Sound Figure 4(b): Structure of Tangible Sound Based on the metaphor of" Source and Drains,' Tangible Sound is made from the following components: two water tanks, a faucet (acting as the source), and four funnels (acting as drains). The physical layout of these components and its view are shown in Figure 4(a) and 4(b). The two tanks are used to generate the flow of water. The first one is on a stand, while the second one is under the stand. The upper tank has a faucet, which makes it possible to regulate the water flow. Within the lower tank, four funnels of different heights serve as drains for the water flow. In addition, a pump is used to transfer water from the lower tank back into the upper tank. The user can divert (touch, scatter) the water, so that some may fall into the main drain (in the lower tank) and some may scatter into the three other drains (funnels). The music volume

Page  00000003 is controlled by the faucet attached to the upper tank. In order to monitor the water flow in the upper level, we measured its diameter under the faucet. Two nichrome wires, measuring the electronic resistance of water, were used for this purpose (Figures 5(a), 5(b)). The same technique was used in the lower level, with a pair of nichrome wires placed in every funnel (Figures 6(a), 6(b)). When the volume of the flow exceeds the draining capacity of a funnel, the level of water within that funnel rises. Accordingly, analog signals are then sent to an i-Cube, a special A/D converter where they are converted into MIDI data, and finally sent to a Macintosh computer via a MIDI interface (Figure 7). water flow's diameter nichrome wires (viewed from under the water) (a) Sensor (b) Figure 5(a): Device of upper flow sensor Figure 5(b): Sensor installation under faucet Two nichrome wires to sense water level (a) (b) Figure 6(a): Sensor for drain Figure 6(b): Sensor installation for drains when the water level in the funnels changes or when the drains receive water flow or drops. Discrete notes are generated by the i-Cube every 20 milliseconds. As discussed in more detail later, this gives the impression that every chunk of water corresponds to one note. In addition, every funnel is assigned to one of four harmonies in an eight-tone scale: " E-flat-major," "F-minor,' "G-minof' and" A-flat-major" The tone scale corresponds to the height of the drain (e.g. the" E-flat-majof' scale is assigned to the highest drain). The user can thus control four kinds of harmonies by choosing funnels of different heights, where each one expresses a unique level of musical tension. Upper Flow User's Input Sensor Velocity Duration Lower Flow Note Height (Table E-flat) -Note Height (Table F-minor) Sensors -f-Note Height (Table G-minor) ) Note Height (Table A-flat) Figure 8: Software structure of note control Sonic Control Configuration. The sonic control configuration monitors the drains in order to select acoustic elements. In this particular configuration, user interaction is captured by computing the difference between the upper flow (under the faucet) and the lower flow (in the main drain). The user can touch, stop and scatter the water flow in order to transform " watef' sounds into " musical' sounds. Furthermore, the user can change the harmony by pouring water in different funnels. Accordingly, an audio sample of flowing water (recorded in a bathroom) is used as a basis to generate the audio output. The MIDI output is composed of five tracks, in which the flowing water sample is played in loops. Time lapses are introduced so that the sample is not played concurrently in different tracks. When the user touches the water flow, the sound is continuously filtered to resonant frequencies of a musical harmony. In other words, " water' sounds are transformed into " musical sounds" according to the following procedure: Every track is associated with one frequency. In other words, the output of the five tracks is a harmony. In every track, the sample sound continuously changes into a band-limited, musical sound by resonance. The frequency of this sound is the same as the track frequency. Every drain is associated with one of four harmonies, namely "E-flat-major" " F-minor' "G-minof' and" A-flat-major'. The height of the drain corresponds to the height of the harmony. When water is poured in a drain, the five sound tracks continuously change to match the harmony associated with this drain. 5. PERFORMANCE Tangible Sound was exhibited during the ATR Open House exhibition, held in November 1999. Many visitors tried the instrument and seemed to really enjoy it (Figure 9). In that sense, our goal to make the production of music an intuitive and enjoyable experience was achieved. An important aspect of the instrument is that it provides a natural tactile feedback when the user touches, scatters and stops the water flow. Spreading water is particularly enjoyable, since it is linked to musical tension. Alternatively, the participant can use a cup to scoop up water from the upper tank. Figure 7: Analog signal sensor' s input to MIDI 4.3. Software System Configuration In combination with the physical configuration of Tangible Sound, we investigated two different software configurations: one for controlling notes and the other for control sounds. These software applications were implemented with the MAX/MSP programming language. Note Control Configuration (Figure 8): The note control configuration uses the direct input of musical elements with MIDI signal control. We used QuickTime Instrument as a MIDI sound generator. This system produces harmonic notes controlled by the flow of water. The velocity and duration of MIDI signals are controlled by the upper flow, which the user controls with the faucet. The level of the lower flow generates both note-on and note-number MIDI signals. For a given funnel, the current level determines the range in which a note is randomly chosen. In other words, if the current level is x, then the note number will be chosen between 0 and 2x. On the other hand, the note-on signals are output either

Page  00000004 Figure 9: Performing Tangible Sound 6. DIscussIoN 6.1. Water as a Tangible Interface Interacting with water can be a multi-sensorial experience, since it is possible to see, hear and touch the liquid. When we designed our instrument, we realized that tactile interaction was particularly interesting. Hence, if the system is to be used by beginners, it is helpful to create alternative representations for music. Based on these observations, we concluded that tangible water was a very suitable and intuitive musical interface. When the user interacts with the faucet, he or she does not directly control the water flow. Moreover, it is not easy to interact with both the water flow and the faucet at the same time. To address this issue, we may later develop an interface to allow the direct control of both Source and Drains. 6.2. Role of Natural Elements in Music Generation Even when the user is not interacting with the water, its flow is constantly changing in very subtle ways. The sounds produced in these circumstances suggest the " personality' of the water. Besides, users never get complete control over the music produced with the instrument. Rather, the music emerges from both human behavior and properties of a natural element. This poetic concept is an important aspect of our instrument. 6.3. Interactive Art and Innovative Musical Instruments In traditional art, a clear distinction is made between artists and spectators. Artists truly interact with the artwork they produce, whether this artwork is a painting, a musical composition or a novel. Spectators, however, are only consumers of this artwork, which they cannot alter in any way. Interactive art aims at extending the scope of traditional art by enabling a new relationship between an installation and its spectators. Indeed, everyone is encouraged to interact and sometimes to transform the artwork. In that sense, spectators become artists and gain a real understanding of the artistic creation. In some respects, Tangible Sound could be considered an interactive art installation. However, we rather see it as a new kind of musical instrument. Indeed, our main motivation was to offer an intuitive interface to support musical composition, even by novice users. Other musical instruments share this goal, such as Theremins and IAMASCOPE (Fels and Mase, 1999). With Theremins, the creation of sounds does not require the performer to touch anything. While this can be amazing for the user, it also makes it difficult to control the music. The interface of IAMASCOPE is also intangible, but visual feedback is provided with the pattern of a kaleidoscope. Here again, the lack of tactile feedback makes it difficult to control the output of the instrument. The two software configurations in Tangible Sound also offer important lessons. As we explained that in the Note Control Configuration, the system conveys the impression that individual notes correspond to " chunki' of water flowing into the drains. The reason for this is that the notes are produced in a discrete fashion. This interesting property was discovered by visitors at the ATR Open House exhibition. In the Sonic Control Configuration, the instrument transforms " watef' sounds into "musical' sounds when the user interacts with the flow. Because this transformation is continuous, the impression of" water chunký' disappears. However, an interesting effect of the transformation of" watef' sounds into " musical' sounds urges the participant to touch the water flow. Based on these observations, we would suggest the design of a new instrument that combines the two software configurations. 7. CONCLUSIONS We have introduced an interactive musical system in which water supports tactile user interaction. The most important aspects of the system are: i) that it makes the creation of music easier and more intuitive, even by novice users and ii) that it generates music from the fusion between human behavior and properties of a natural element. This system will ideally find applications in domestic settings and living environments, such as: 1) kitchens and bathrooms, where "musical faucetý' would allow people to play music while they wash dishes or take showers; 2) outdoor entertainment locations, such as pools, rivers or lakes, where playing music would become an additional distraction; 3) schools, where the instrument would make music education more enjoyable. As these examples suggest, we believe that the main applications for Tangible Sound will be associated with enjoyment and entertainment. Our guess is that it will become a device with which users interact in very affective ways. This hypothesis was indeed verified by observing the reactions of people trying our first hands-on installation. ACKNOWLEDGEMENTS The authors would like to thank Ryohei Nakatsu for his support and Timothy Chen, Kazushi Nishimoto, Rodney Berry, Olivier Liechti, and Hisashi Matsuse for invaluable discussions on this project at ATR. We would also like to thank the members of Keio University7 s Iwatake Lab and Yasumura Lab, especially Prof. Toru Iwatake and Prof. Michiaki Yasumura. Finally, we appreciate Atau Tanaka and Edwin van der Heide, sensor-band specialists, for discussing the primitive ideas of this work. REFERENCES [1] Sugihara, Y." Water Display," Conference Abstracts and Applications SIGGRAPH99, pp. 183, 1999. [2] Ui, N. and Setoh, K., " Wave Rings," http://www.sfc.keio. ac.jp/~noriyuki/pcs, 1998. [3] Ishii, H. " Tangible Interfaces,' Conference Abstracts and Applications SIGGRAPH99, pp. 127, 1999. [4] Sakonda, N. " Water Machine," Bumpodo Gallery, 1998. [5] Fels, S., and Mase, K. " Iamascope: a graphical musical instrument," Computers & Graphics 23, pp. 277-286, 1999. [6] Yonezawa, T. and Mase, K. " Interaction of Musical Instruments Using Fluid,' Transactions of the Virtual Reality Society of Japan, Vol. 5, No. 1, pp. 755-762, 2000. (in Japanese)