Page  00000117 THE TONCOLARIUM - IMMERSIVE COLOR-MUSIC INSTRUMENT David Yanagisawa Interactive Sound Design, Kristianstad University, Sweden dykun@brikks.com www.interactivesound.org ABSTRACT The Toncolarium is a hybrid. Built around a water-filled transparent acrylic glass container, it resembles an aquarium. It combines colors and tones, light and sound and connects elements of two worlds - the real and the virtual. The background of this artifact is an attempt to follow the old tradition of experimental color-music instruments and to explore the phenomenon of synethesia. Throughout the centuries artists and instrument makers have come up with many ideas on how to integrate music and colors. Among those instruments are the famous color organs, which formed an earlier tradition of visual music. The technological achievements in our modern days have made it possible to build advanced musical instruments that may integrate, generate and communicate colors and sounds in order to simulate a tone-color synesthesia. Probably such instruments can give greater inspiration to music composers and a deeper level of satisfaction for the performers of music. 300 years of experiments may only have brought us to the dawn of the color-music instrument making. 1. INTRODUCTION To approach the main concept of this contribution, a short historical survey as well as some sources of inspiration is provided. The French Jesuit monk Louis Bertrand Castel, credited to be the father of color-music, was inspired by Newton's theory that musical and color harmonies are related by means of the frequencies of light waves and sound waves. In 1725 Castel started to develop a color harpsichord in collaboration with the instrument maker Rondet. The instrument was called Clavecin Oculaire and had paper strips that appeared on top of the instrument when a key was pressed. The paper strips were lighted by candlelight [1]. The term "Color Organ" was first used in a patent application by Alexander Wallace Rimington in 1893. His three-meter-high color organ resembled a customary house organ with a cabinet of fourteen colored lamps on top. The light of the colored lamps could be adjusted to certain gradations of hue, brightness and saturation. This meant a tremendous progress compared to the paper strip harpsichords of the earlier centuries. Like most color organs, Rimington's instrument did not produce musical sounds. The color organ had to be played simultaneously with an organ that produced musical sound [2]. Other influences in the field of color-music are Bainbridge Bishop, Alexander Scriabin and Adrian Bernard Klein. Many interesting ideas of musical color scales have come up among different, often unrelated persons. As we can see in Figure 1, for the most part, every one felt that the note "C" was red. C C D D* E F sarac, Newtom 1704::Louise Bertrand Casd 17-4 AYa 1: bc R R1ml: n,to n -1:89 Ri~nibrdge Bshop 1:893 Aexander Bcriabin 1911 Adaria Bernard. Klein 1930 F G AA Figure 1. Color Scales. 1.1. Synesthesia The term "synesthesia" means "senses coming together". The most prevalent form of synesthesia is the phenomenon of seeing colors when hearing music, named audition coloree by the French psychologists [3]. It is estimated that synesthesia may be as prevalent as 1 in 23 persons across its range of variants [4]. A wellknown early mention of sound-color synesthesia can be found in John Locke's, An Essay Concerning Human Understanding (1689). He tells about a blind man who made use of the explication of his books and friends and finally understood what scarlet signified. He said, "it was like the sound of a trumpet" [5]. There is rarely agreement amongst synesthetes that a given tone will be a certain color. However, consistent trends can be found, such that higher pitched notes are experienced as being more brightly colored. Also in our everyday language we have these common sensory associations. Musical tones can be high or low, and they can be dark or light, even straggly. Rock music can be both heavy and hard. Jazz can be cool and positive. A tone can be rich and warm or cold and metallic, just as a drawing can be made of warm colors and another made of cold colors. 1.2. Aims The question here is: What makes certain combinations of colors and tones (sounds) more pleasing to us than others? Of course, this is also a question about culture. Music in the far-east is structured very differently than in the west. This is also true of color. Combinations of color that are found in places like India or Africa are 117

Page  00000118 very different than that of central Europe. To keep it simple, the decision is to focus on western culture and music, and to approach the matter in a practical manner. The core aims of this work include the following main activities: 1) build a color-music instrument, 2) use computer software to translate the musical information to colors. 2. CONCEPT The main approach was to build an instrument around a physical substance that could more naturally integrate sounds and colors and let the performer easily change both color and tonal scales. Instead of playing on a plain keyboard and look at a flat screen, the player shall be able to feel the depth of the sounds and the ocean of colors. Water may here be considered as an interesting medium. 2.1. How to colorize water? To use ink or water-color to colorize water could be an interesting effect, but the water would quickly be turbid and darkened. Contrary to painters who make use of physical paint, music composers always deal with an immaterial "substance". Music is therefore often viewed as the highest art. But colors in the form of light approach the immaterial state and thus come very close to the realms of sounds and music. Colors in freed form will also introduce mobility in time, rhythm, and combination, slow or rapid and varied. When Sir A. Wallace Rimington introduced his color organ, he said: "There has, in fact, been no pure colour art dealing with colour alone, and trusting solely to all the subtle and marvellous changes and combinations of which colour is capable as the means of its expression. The object of the present invention is to lay the first stone towards the building up of such an art in the future. " [6] Therefore a decision was to let colors in the form of light stream up into the water from underneath. High intensive LEDs with wide angle dispersion are now available. LED is a very efficient light source. They are small and have a long life. They are here to revolutionize the color-music field, by being incorporated into the music instruments. The Toncolarium make use of 36 full-range RGB LEDs (9400 mcd) with an angle dispersion of 40~. The next challenge was to find a way to let the light spread in the water container and diffuse nicely. Acrylic glass will naturally reflex and spread the light in a very beautiful manner. It has optical qualities with unsurpassed transparency and high light transmission. 2.2. The Toncolarium The name Toncolarium reveals the basic concept of this instrument - the relationship between musical tones and colors, known as tone-color synesthesia. The ending part -arium refers to a box or container that holds something - in this case a concept of color-music. It can also be associated with the term aquarium. Like an aquarium, the Toncolarium is filled with water and that plays a vital role for it in order to work. Figure 2. The Toncolarium. The Toncolarium is made of transparent acrylic glass. The size is 100*250* 1000 millimeters and the container is filled with water. The instrument is controlled by four short hoses made of transparent PVC tubing. A metal thread is placed inside each hose, from the bottom and all the way to the top, where a cable connects it to a microprocessor underneath the instrument. The microprocessor reads the data from the hoses and the 12 metal contacts which are placed at the bottom of the water, one in each row of LEDs. The distance between the edge of a hose and a metal contact at the bottom determines the amount of conductivity. Ordinary tapped water gives a suitable resistance. By putting a hose into the water, LEDs underneath the water are highlighted with specific colors. The player controls the amount of light by how deep a hose is immersed. The deeper the stronger the light will be, causing louder volume to have a brighter light. The flexible PVC hoses can be placed and hold at a certain depth. Each of the twelve rows of LEDs is controlled from a computer, so the color for that row can be changed to any color-value in the RGBscale instantly. A sound is played when a hose has contact with the water. The area above each of the twelve rows can be assigned to trigger a unique sound. Each one of the four hoses can activate any of the twelve contact areas and all four hoses can be used simultaneously, making it possible to create more complex chords. The player controls the volume of a sound by how deep a hose is immersed. The deeper, the louder the sound will be. 2.3. Target groups The Toncolarium is meant to create an ambient atmosphere and stimulate the color-musical sensation. According to Scriabin the presentation of the right color corresponding with music works as "a powerful psychological resonator for the listener" [7]. 118

Page  00000119 2.3.1. A music instrument 2.5.2. Colors: The target in focus here is first of all the performer. The main purpose of this project is not to bring a spectacular lightshow to an audience, but to give the performer or composer a stimulating approach to the instrument in order to enhance creativity. Matching colors will here enrich the improvisation and composing phase in several ways. Just like how new sounds, chords, tensions or melody phrases inspire and open up new possibilities, colors in the form of light will help and guide the improviser/composer to find new ways to combine the elements, combinations that he had not thought of before. 2.3.2. An interactive installation A light atmosphere which has been formed by colors can help people to relax, so the Toncolarium can also be used for color and sound therapy. By letting algorithms control the changes of sounds and colors automatically, the Toncolarium could be a perfect artifact which will reduce tension in for example a waiting room. 2.4. Interaction The Toncolarium has two basic operation modes: instrumental and musical. In instrumental-mode, each of the twelve contact areas has a color that corresponds to a tone. Both the tone and color scale can be changed in the computer program so the synesthete can tune the instrument to fit him personally. This can be down manually or by letting complex algorithms automatically control the relations between pitch, amplitude, color and light intensity. In musical-mode, the instrument plays prerecorded sound loops and acts like a sound mixer. For example, in sea-program the instrument produces blue-green colors that slowly move from one side to the other like waves. When a performer puts a hose into the water the LEDs at that place will change color to red and a sound corresponding to that place will start playing in a loop. Both track volume and light intensity are controlled by how deep the hose is immersed. 2.5. Output and input Two microprocessors handle the inputs and outputs. A second microprocessor is used to output values from the computer program to the twelve rows of LEDs. The colors can be changed easily from the computer program. Since every LED is a full-range RGB LED, it's possible to generate dynamic changes of colors, not only in the horizontal position but also in the vertical position of the instrument. 2.6. Computer animated colors and graphics A second aim was to use computer software to translate the musical information to colors on a computer screen. A short technical description of how the system works is here provided. The incoming data from the Toncolarium (depth value and tone number) are sent from Max/MSP to Processing by using the Maxlink. Processing is used to generate all the computer graphics and colors displayed on the screen. Each of the four PVC hoses controls a color object (Figure 3). Figure 3. The melodies are recorded as both midi and audio files. Midi velocity values are connected to y-position variables in the Processing sketch and midi pitch values are connected to x-position variables. The audio signals are converted to floats in Max/MSP and control the size of the objects appearing in the Processing window. See example video file "4Suns. mov " [8]. 2.7. Computer systems and equipments Max/MSP 1 Processing 2 Maxlink library 3 Arduino ' Other: Soundcard, MIDI Keyboard, Laptop and Projector 2.5.1. Sounds: The data from the instrument (depth value and tone number) is sent to a microprocessor underneath the instrument. The microprocessor is connected to a computer via USB. The incoming values are converted to midi data in a computer program. In instrumentalmode, sounds are produced by virtual midi instruments. In musical-mode, the sounds come from audio files especially composed for the show. 1 A graphical environment for music, audio, and multimedia by Cycling74. http://www.cycling74.com/ 2 An open source Java based programming language for images, animation, and interactions. http://www.processing.org/ 3 The MaxLink Java libraries enable communication between Processing and Max/MSP. http://jklabs.net/maxlink/ 4 An open-source physical computing platform http://www.arduino.cc/ 119

Page  00000120 3. CONCLUSION The idea to make use of water as a medium to generate music isn't new. There has also been many artist and musicians who have been experimenting with music and colors. The Toncolarium is a mix of both these concepts. The idea to use water as a medium for signals controlling audio and light has been tested, and the result is very satisfying. Ordinary tapped water is a sufficient conductor for this purpose. There is no need to add chemicals or salt to the water. The Toncolarium is built around a small, fast and affordable (opensource) physical computing platform, Arduino. The Toncolarium incorporates two of them, one for input signals and a second to control the small high intensive LEDs, which are being better and better and are a perfect match for a color-music instrument. It is possible to control both sound and color output in one program, Max/MSP, making it easy to try new combinations of tones and colors. In instrumental-mode, the instrument is limited to twelve tones and colors. Twenty-four rows would have made the instrument even more musically playable and attractive. But the instrument would also have been longer. However, the Toncolarium is only a prototype and might expand in the near future. Computer animated colors and graphics in Processing which react on musical information have also been tested. An example is provided which shows how both midi and audio information can be used to control the shape and motion of colors, and how colors can be blended and added together to form abstract multi colored patterns. As a music instrument, the Toncolarium is meant for experimentation only, and is not meant as being a final solution. As an interactive installation, it will do a great job to create a relaxing atmosphere and reduce tension in for example a waiting room at the dentist or physician. 3.1. Future development For future developments my intention is to try to give answers to the following questions: How can color and sound interact and be perfectly integrated into one single instrument in order to materialize a satisfactory analogy between color and music? How would a color-music instrument for the 21-century look like? How about a synthesizer or digital audio workstation that besides generating sounds, displays colors in realtime and lets the composer create his own shapes and colors for each sound or track and control their motion in the sequencer, just like midi and audio. How about translating notes, chords and rhythmical patterns to colors and shapes, and to be able to print them just like scores? There can be several practical advantages to use colors and shapes in combination with music. For example, it can help students to learn the language of music quicker by memorizing the relationships of the colors. Music instruments with built in LEDs can stimulate students at the first learning phase, and help them to quicker understand where the tones on the instrument are placed and how to find them. The Interactive Sound Design program at Kristianstad University has made it possible for me to explore this fascinating field of science which needs a rebirth and there still remains a lot to learn. I would like to thank electro technician Ake Bermhult for invaluable help with the prototype. My hope is that this project will continue and that my experiences will be shared at the ICMC 2007 conference and performances. 4. REFERENCES [1] K. Peacock, "Instruments to Perform Colormusic: Two Centuries of Technological Instrumentation", Leonardo 21, 397-406 (1988). [2] K. Peacock, "Instruments to Perform Colormusic: Two Centuries of Technological Instrumentation", Leonardo Vol. 21, (1988), pp. 397-406. [3] Cretien van Campen, "Artistic and Psychological Experiments with Synesthesia" Leonardo, Vol. 32, No. 1 (1999), pp. 9-14. [4] Simner, J.; C. Mulvenna & N. Sagiv et al. (2006), "Synaesthesia: The prevalence of atypical cross-modal experiences", Perception 8(35): 1024-1033. [5] John Locke, "An Essay Concerning Human Understanding", Book III, Chapter IV, section 11. [6] "Colour Music, the Art of Light", by A.B. Klein, Lockwood, London, 1930, pp 256-261. [7] K. Peacock, "Synesthetic Perception: Alexander Scriabin's Color Hearing", Music Perception 2, No. 4, 483-506 (1985). [8] Yanagisawa David, "4Suns.mov", example video file, http://www.myspace.com/dykun 120