Page  1 ï~~DESIGNING NEW INSTRUMENTS FOR THE HUMAN TOUCH - CASE SORMINA Juhani Raisanen University of Arts and Design Helsinki, Media Lab Voudinkuja 3 B 8, 02780 Espoo, Finland p. +358 40 5227204 Juhani.Raisanen@taik.fi ABSTRACT Musical instruments are cultural objects that constitute a substantial part of traditional musical praxis. If new innovations in this field wish to survive, they should respect the history. At the same time, they still have to appeal to contemporary musicians, composers, and audience. In this paper, one proposition for such an innovation is made. This paper describes the Sormina, a new virtual and tangible instrument, which has its origins in both virtual technology and the heritage of traditional instrument design. Instead of being just a collection of control knobs, sormina aims to be a solid physical instrument, ready to be utilized in modern classical music. This is accomplished by using a fixed set of three layers in the construction. Firstly, much attention has been put to the design of the interface, the touch, but two other layers have also been crucial: the sound synthesis software and the mapping of the control parameters. Evidently, the amplifier and loudspeakers form also an integral part of the instrument. 1. INTRODUCTION Sormina is a new musical instrument that has been created as part of a research project in the University of Arts and Design Helsinki, Media Lab. Sormina uses sensors and wireless technology to play music. Its design is guided by traditional instrument building. In new wireless technology, the instrument loses part of its traditional character. The physical connection between the sounding material and the fingers (or lips) is lost. The material does not guide the design, which puts the designer in a totally new situation with new questions. This study tries to answer these questions by exploring the design of a new instrument that is intended for use in the context of a live symphony orchestra. The research has started from the concept of the interface, which traditionally is held in hands or put in the mouth. The playing posture of the musician, the delicate controllability of the instrument and the ability to create nuances are considered as the key phenomena of the new design. Visual aesthetic and usability are of equal importance. 2. MOTIVATION The motivation for this innovation is the desire to create totally new musical instruments in the context of classical music by using computers and sensors. We are interested in designing digital instruments that could be accepted as part of the standard symphony orchestra. We believe that classical music can benefit from the current developments in digital technology. The symphony orchestra has been quite stable during the last century, although there have been some experiments using electronics. Sormina aims to encourage the symphony orchestra to develop further to meet the challenges of the digital era. This project participates in a long tradition of similar innovations, starting from the Theremin, which is a rare example of a musical innovation to become part of classical music practise. In addition to Theremin, one of the most influental to the current research has been Rubine and McAvinneys article in Computer Music Journal 1990, where they presented their VideoHarp controller and discussed issues related to its construction [1]. Also Michel Waiswisz and his Hands has been a great inspiration [2]. Recently, Malloch and Wanderley have proposed the Tstick [3]. Important questions concerning parameter mapping have been discussed in Hunt, Wanderley and Paradis [4]. 3. CONSTRUCTING THE INSTRUMENT 3.1. Hardware, potentiometers Structurally, the sormina is built using a Wi-microdig analog to digital encoder, a circuit board for the wiring, and 8 potentiometers with custom-made, wooden knobs. The Wi-microDig is a thumb-sized, easily configurable hardware device that digitizes up to 8 analog sensor signals then transmits these messages wirelessly to a computer in real-time for analysis and/or control purposes [8]. The custom-made circuit board takes care of the wiring. The potentiometers are mounted in the circuit board in an upright position, and the encoder unit is also attached to the circuit board. The knobs of the potentiometers are arranged in a straight line on top of the instrument. The construction of the controller is open: it is not put in a box or cover. With the help of this arrangement, the visual design appears light and spacious. However, the decision to use no cover is subject to change in the forthcoming prototypes, as the openness makes the construction vulnerable to dust and moisture. The sormina makes use of 8 potentiometers, which is the maximum number of sensors to be connected to the encoder. The choice between sensors or potentiometers was made on the basis of three main arguments: stability, precision and tangibility.

Page  2 ï~~The usual sensors do not preserve their state when not touched, whereas a potentiometer changes its state only by intentional action. Potentiometers proved to be very precise compared to many sensors. The author was also fond of the tangibility of the knobs. The selection of wood as the material for the knobs was made for the purpose of the feeling in the fingers of the musician, her touch. They feel warm and soft in the fingertip. Another argument was the visual esthetic character of the material. Human engineering issues, or ergonomics, have been a substantial part of the decisions made with the design. The wirelessness helps making the interface light, small, and hand-held. The knobs fit exatcly the dimensions of the fingers of a human being. Also the motion of the potentiometer is well fitted with the trajectory of the fingers. The visual interface comprises sliders, number boxes, and basic notations for the potentiometer input. At the same time the interface is capable of recording a control sequence, which was found useful for learning to play the instrument. While the recorded sequence is playing back, the visual information about the state of the potentiometers is shown on the interface. The sound is created using a sound synthesis patch created for the Reaktor software. The patch allows the control of several features of sound synthesis. The mapping of the potentiometers to the sound synthesis software appeared to be of crucial importance. It was decided that there should be 8 potentiometers, which was found to be a very useful restriction. It was assumed that a human being cannot handle too many controls at the same time. Too many options could result in indeterminacy. Also, with 8 potentiometers, nearly all of the fingers could still be utilized for controlling purposes. Figure 1. Sormina is a virtual instrument with wooden knobs. 3.2. Software The software for sormina has been programmed using Max/MSP and Reaktor. It consists of three parts: one handles the communication with the encoder through bluetooth, the second takes care of the user interface, and the third produces the sound. In addition, external software, Sibelius, was used for the notation. The Wi-minidig comes with its own software, which actually is not used in this project. This software is meant to take care of the bluetooth connection and let the user decide the interpretation of the sensor data, which is then sent forward as MIDI information. In addition to this rather laborious software, the company alsso offers on the web site for the same purpose a Max/MSP patch, which proved to be handier for the purpose of the project. The wi-microdig patch for Max/MSP appeared to handle the communication through bluetooth with the encoder quite reliably. The Max/MSP programming environment was also favored for its usefulness in other parts of the project. A visual user interface was programmed using Max/MSP, which also handles the connection to the encoder. One purpose of the interface is to give the musician visual cues in controlling the instrument. This proved to be beneficial especially in the learning phase. In addition to the feel in the fingertip, it was helpful to see the state of all the potentiometers at one glimpse on the screen. Figure 2. Part of the visual interface The Reaktor software was chosen as the sound engine for the project, although the use of two different pieces of software instead of only one has its drawbacks. Reaktor was found to be more amenable than Max/MSP for the purposes of this project. The sound synthesis patch in Reaktor, created by Martijn Zwartjes, comprises a 96-voiced noise generator with filters and reverb. The patch has 26 controls for mapping but because of the restrictions of the hardware, only some of them were possible to choose. One solution for the mapping problem could have been to

Page  3 ï~~use one potentiometer for several controls on the sound software but it was found that this would be unwise on a large scale, although some potentiometers are connect to two parameters. 3.3. The selection of the parameters The digital sound synthesis used in sormina copies an analog synthesizers subtractive synthesis. White noise is used as a basis waveform, which is then filtered through time-variable filters to achieve usable timbres. Adding reverberation to the signal gives the musician a large controllable variation of the sound. All the parameters used in sormina control either the time-variable filters or the reverberation control. The choice was accomplished by testing several combinations of the original 26 parameters. After longterm experiments a choice was made that includes 5 time-variable filter parameters, 2 reverb parameters, and one for the overall volume (see Figure 3). 3.4. Notation One important part of the new instrument design was the attempt to notate the music created with the sormina. It was challenging to put up a link with Max/MSP and notation software for notating eight parameters in the The Sibelius software was chosen for this purpose. The note heads were changed to triangles in order to distinguish them from normal pitched notes. A number was added near the note head to be more precise. With the possibility to create a musical score, it is possible to preserve the music also in a conventional way. The notation is also a way to gain composers interest in the instrument. The zero position of the controller was put in the second ledger line below the staff, and the highest position, 127, above the staff. This way the notation appeared very readable. Numbers were appendend to the noteheads for the need of exactness. 4. IN PERFORMANCE Much of the development of the sormina has been conducted through collaborations with other musicians. The sound created by the instrument seems to fit quite well with string instruments, especially the cello. The reason for this fact was considered to be the use of noise generators as the main sound source. The sound of acoustic instruments has many characteristics of white noise. Singing voices showed a similar resemblance to the sormina sound, also. 4.1. Concerts same score. 1: Attack: Resotn 3: Density 4: Ro om 5: Decay 6: Pitch 7: Reverb 8: Vume There have been several public concerts during the first year of the instrument's existence. In addition, the sormina has been presented to researchers and students,........................ and in seminars. In November 2007 the sormina was played with the chamber choir Kampin Laulu (see Figure 4).. i 0::; _<,<ti<....:::::.t::::........... ate...,.::: a::: I vi ------------------------------------------------------------............................................................................... t. Figure 3. Notation of the parameters of the sormina Figure 4. The author playing the instrument in a concert

Page  4 ï~~The last performances of 2007 were in December in Los Angeles, where the instrument was being presented for the art students in the California Institute for the Arts, Calarts. Two concerts were also given in art galleries and jazz cafes in the area. 5. DISCUSSION The aim of the sormina project was to explore the main principles of the instruments in classical music, from the musician's point of view, and with these findings to create a new, stable electronic music instrument that could be accepted in a symphony orchestra. The results suggested the importance of three layers in the design of new instruments. The first layer is the sound-synthesis that defines the audible response. The second is the mapping of the gestures to the sound parameters, which constitutes the instrument in a conceptual manner to the musician. The third layer, often overlooked in the creation of new digital music instruments, DMIs, is the materiality and usability layer of the controller. Much weight in the research has been put to the human hand and its capabilities. The author has followed Curt Sachs' findings about the hands and feet being the first intstruments [5], and Malcolm McCullough, as he praises our hands as a best source of personal knowledge [6]. A remarkable source for understanding the importance of music playing has been Tellef Kvifte, who formulates a classification of instruments using playing techniques, not based on the way the musical sound is produced [7]. The sormina research suggests that the touch and feel of the interface is important to take into account when designing new instruments. The musician uses subtle, almost intuitive and unconscious movements of her body. The fingers, for example, have developed through evolution to take care of the most sophisticated and precise actions. Therefore it is reasonable to use the fingers for playing music. In the culture of the human being, the fingers have been crucial for surviving. Even today, they are used extensively, to express our thoughts, by writing with a pen or a computer. In the course of history, traditional instruments have matured to be well adapted to the human body. Their long evolution has given them power to survive even in the era of computers. Through careful examination of their principles, it is possible to learn from their pattern and use the results in the design of totally new electronic instruments. In the present research, the role of the physical interface has been found to be fundamental for such a design. It appears that attention should be paid to the physical appearance of the instruments in order to build stable instruments. Sormina aims to be more than a controller. As Rubin and McAvinney formulate, a musical instrument may be thought of as a device that maps gestural parameters to sound control parameters and then maps the sound control parameters to sound [1]. By binding together a fixed set of sensors with a stable sound source, we have developed sormina into an instrument, not a controller. Sormina attempts to be engaging to new musicians, but also rewarding for the professionals. Based on the current evidence, these goals have been reached to a large extent. 6. FUTURE DIRECTION The current research has used the observation of traditional musical instruments and their user experience for the design of a new electronic music instrument. Still, the scope of the exploration has been narrow, concentrating primarily on the author's experience of acoustic instruments. In the future, a more systematic inquiry will be accomplished, where professional musicians will be observed and interviewed about their playing habits. Also, perceiving the learning process in the study of classical music instruments can reveal qualities that could then assist in new instrument design. The instrument has its own web site where the most fresh developments and recordings are being published [9]. 7. REFERENCES [1] Rubine, D. and McAvinney, P. "Programmable Fingertracking Instrument Controllers". In Computer Music Journal, Vol 14, No. 1, Spring 1990, 26-40. [2] Waisvisz, M. "The Hands, A Set of Remote MIDI Controllers". In Proceedings of 1985 International Computer Music Conference. San Francisco, 1985 [3] Malloch, J. and Wanderley, M. "The TStick: From Musical Interface to Musical Instrument". In Proceedings of the 2007 Conference on New Interfaces for Musical Expression (NIME-07), 2007, 66-69. [4] Hunt, A. and Wanderley, M. and Paradis M. "The importance of parameter mapping in electronic instrument design". In Proc. of the 2002 Conf on New Interfaces for Musical Expression (NIME-02), 2002, 149-154. [5] Sachs, C. The History of Musical Instruments. Norton, New York, 1940, 25-26. [6] McCullough, M. Abstracting Craft. The Practiced Digital Hand. The MIT Press, Cambridge, Massachussets, 1996, 1-15 [7] Kvifte, T. Instruments and the electronic age. Toward a terminology for a unified description ofplaying technique. Solum fdrlag, Oslo, 1988, 1. [8] Wi-Microdig v6.00/6.1 technical description. <http:/iit ic nsovyst mscom!ctal &o a. shp/pagesid 1i53>. Infusionsystems.com. [9] The address of the sormina web site is <lit orl-ni na C (.)III>