Page  00000001 WHAT I S THE C OL OR OF THAT M/USIC PE RF ORMANCE? Roberto Bresin KTH - Royal Institute of Technology, Dept. of Speech Music and Hearing Stockholm, Sweden ABSTRACT The re-presentation of expressivity in music is still a fairly unexplored field. Alternative ways of representing musical information are necessary when providing feedback on emotion expression in music such as in real-time tools for music education, or in the display of large music databases. One possible solution could be a graphical non-verbal representation of expressivity in music performance using color as index of emotion. To determine which colors are most suitable for an emotional expression, a test was run. Subjects rated how well each of 8 colors and their 3 nuances corresponds to each of 12 music performances expressing different emotions. Performances were played by professional musicians with 3 instruments, saxophone, guitar, and piano. Results show that subjects associated different hues to different emotions. Also, dark colors were associated to music in minor tonality and light colors to music in major tonality. Correspondence between spectrum energy and color hue are preliminary discussed. 1. INTRODUCTION Studies on synesthetic experiences related to music perception have reported non-consistent association between music, usually music events, and colors [1]. We want to investigate if these differences could be explained in terms of expressive contents of music performances and consequently if we could use color as one indicator of expressivity in music performance. This work is part of a research project 1 that aims to develop a computer system for teaching students to play expressively. Part of the system is a tool for automatic extraction of acoustic cues that are considered important for the analysis of performance expression. These cues include duration, sound level, articulation, and vibrato. For example, staccato articulation may be typical of a happy performance, while legato articulation may be typical of a tender performance. This tool was presented at ICMC 2002 in G~teborg [2]. It is important that music students are trained to control acoustic cues eflfectively to achieve different expressive performances. Results from analysis of acoustic cues can be presented to the performer in a variety of ways, for example in term of graphs or tables representing mean values and deviation curves after a post-processing of the 1 Feedback-learning of Mlusical Expressivity (Fee-Me): http://www.psyk.uu. se/hemsidor/musicpsy/ performance. This can be useful for a more analytical understanding of what the performer did. A different display is needed in the case of a system that provides real-time feedback of musicians' expressive intention. In order to map intended emotions a visual display was designed in which emotions are mapped to colors. For a meaningful association of intended emotions in music performance to colors, an experiment was needed. This is presented in the following sections. 2. MIETHOD An experiment was designed in which it was tested how subjects associated different emotionally expressive performances to diflferent colors. The experiment was conducted with the help of a specially designed computer -program. 2.1. Music examples For this experiment performances of two melodies were used, Brahms' 1st theme of the poco allegretto 3rd movement Symphony Op.90 No.3, in C minor, and Haydn's theme from first movement of Quartet in F maj or for strings, Op. 74 No. 2. In a previous experiment [4] the two melodies were performed with piano, guitar, and saxophone by nine professional musicians, three musicians for each instrument. The musicians were asked to perform the two melodies with twelve different emotional intentions, namely happiness, love, contentment, pride, curiosity, indifference, sadness, fear, shame, anger, jealousy, and disgust. In a listening test, subjects were asked to rate these performances with regards to their emotional character. Analyzing the results from this listening test it emerged that some performers were better in communicating certain emotions. The performances that received the highest rating for each instrument and each emotion were chosen for the present experiment. Thus the musical material for the experiment consisted in twelve performances for each instrument and each melody (12 x 3 x 2) for a total of 72 performances. 2.2. Colors Colors were used for rating each performance. Colors were coded using the Hue Saturation Brightness (HSB) scheme 2. Hue is the actual color. Saturation is the purity 2 See http:/ /www.cecs.csulb.edu/ ~j ewett/colors/ hsb.html

Page  00000002 of the color and is measure in percent. Brightness is measured in percent from black (0%) to white (100%). The colors used were red, orange, yellow, green, cyan, blue, violet, and magenta, and their bright and dark versions, for a total of 8 x 3 colors (see Table 1). Low saturation and low brightness were coded with value 0.5 (50%). High saturation and high brightness were coded with value 1 (100%). For a list of all the colors used in the experiment see Table 1. Combinations with low percentages of both saturation and brightness, and the use of 0% for saturation or brightness, were not included since they led to meaningless hues or to black and white. The computer screens were calibrated with Pantone Spider to a luminance of about 90 cd/m2 2.3. Subjects Two groups of subjects took part to the experiment. Group 1 consisted of eleven students of psychology, 7 females and 4 males, aged 21-49. Group 2 was composed of researchers in and students of speech and music acoustics, 5 females and 6 males, aged 24-42. 2.4. Procedure Colors were presented on the computer screen with the help of a computer program, COLORANCE (COLOR and performANCE perception test), specially designed for this Color Hue Saturation Brightness red 0 1 1 dark red 0 1 0.5 light red 0 0.5 1 orange 0.083 1 1 dark orange 0.083 1 0.5 light orange 0.083 0.5 1 yellow 0.166 1 1 dark yellow 0.166 1 0.5 light yellow 0.166 0.5 1 green 0.333 1 1 dark green 0.333 1 0.5 light green 0.333 0.5 1 cyan 0.5 1 1 dark cyan 0.5 1 0.5 light cyan 0.5 0.5 1 blue 0.666 1 1 dark blue 0.666 1 0.5 light blue 0.666 0.5 1 violet 0.749 1 1 dark violet 0.749 1 0.5 light violet 0.749 0.5 1 magenta 0.833 1 1 dark magenta 0.833 1 0.5 light magenta 0.833 0.5 1 Table 1. List of colors used in the experiment and their combination of hue, saturation and brightness experiment. COLORANCE presents 8 colors at a time to the subject, with the same saturation and brightness values. Subjects were instructed to judge how well different colors fit a particular music performance by rating each of 72 performances using the color palettes shown on the computer screen. For each performance three color palettes were presented, one for each combination of hue, saturation, and brightness, as presented in Table 1 (i.e. one palette for default colors, one for dark colors, and one for light colors). The subject could listen to a music performance as long as she liked to and rate it with the sliders associated to each color corresponding to a scale from 0 to 10. If the subject did not like a specific color, she could hide it by clicking on it, and the color was rated with value 0. The program window covered the entire screen so that it was the only visible window to the subject. The order of the performances was randomized for each subject. The order of the three-color palettes was also randomized for each subject, as well as the placing on the screen of the colors forming a specific color palette. Colors with same hue appear in randomized positions in color palettes with different combinations of saturation and brightness, i.e. the red color will appear in different positions for default (saturation 1, brightness 1), dark (saturation 1, brightness 0.5) and light (saturation 0.5, brightness 1) color palettes. Group 1 listened to Brahms' melody and Group 2 to Haydn's. 3. RESULTS The emotional intention of the performances (the independent variable) had a considerable effect on the listeners ratings of the performances (the dependent variables hue, saturation and brightness). An analysis of correlations between emotional intentions in the performances and ratings of hue, saturation and brightness, reveals some significant correlations, as Emotion Hue Saturation Brightness Happiness -0.25 -0.04 -0.19 Love -0.08 -0.28 0.68* Pride -0.20 0.07 -0.30* Tenderness 0.10 -0.09 0.39* Curiosity -0.10 0.03 0.24 Contentment -0.08 -0.17 0.31* Anger -0.19 -0.44* 0.04 Sadness 0.16 -0.15 0.47* Fear 0.19 0.06 0.38* Disgust 0.30* -0.25 0.20 Shame -0.03 -0.32* 0.57* Jealousy 0.09 -0.12 0.29* Table 2. Correlations between color components (hue, saturation and brightness) and emotional intentions in the performances. Correlations marked with * are significant at p <.05

Page  00000003 shown in Table 2. Significant correlations were found for brightness and the expressive performances performed with love, pride, tenderness, contentment, sadness, and fear. Also significant was the correlation between saturation and the intentions anger and shame. There was only one significant correlation for hue, with disgust. In the following a more detailed analysis of the effect of the hue, saturation and brightness is presented. 3.1. Hue If we look in detail to the average color profile for each instrument and each emotion, we find that saxophone and guitar have very similar profiles within the same melody and emotion. As an example see Figure 1. For saxophone and guitar the dark versions of colors red, orange, blue, violet received higher average preferences, while for piano performances their light versions get higher rates (except for violet). The color profiles for saxophone and guitar differ from those of piano, probably because its lack of possibilities of controlling timbre features such as attack time and vibrato. In fact these are acoustic cues that are important in the communication of emotions, e.g. a slow attack time and vibrato are characteristic of a performance SHAME Emotion Hue Happiness 0.167 (Yellow) Love 0.667 (Blue), 0.75 (Violet) Pride 0.167 (Yellow) Tenderness 0.75 (Violet) Curiosity 0.5 (Cyan) Contentment 0.083 (Orange) Anger 0 (Red) Sadness 0.75 (Violet) Fear 0.667 (Blue) Disgust 0.75 (Violet) Shame 0.083 (Orange) Jealousy 0 (Red) PIANO Haydn I Brahms GUITAR- - - -Haydn] Brahms SAXOPHONE' [rahs Table 3. Hue values that received highest mean rating for each emotional expression portraying sadness while fast attack time is typical of angry performances [3]. In the case of "shame", if we consider only the hue parameter than orange is the color mostly associated to this emotional intention. Similarly, table 3 presents the hue values that received highest mean rating for each emotional expression. Results were independent from the melody while some differences emerged between instruments. 3.2. Saturation A general observed tendency for saturation is that the minor tonality composition (Brahms) tends to be associated to colors with higher saturation (dark and default colors). This tendency is independent from the instrument. When taking into account also the instrument variable, it emerges that Brahms melody ratings follow a different trend for piano, for which colors with lower saturation level seem to be preferred. 3.3. Brightness A general observed tendency for brightness is that the minor tonality composition (Brahms) tends to be associated to dark colors, and the major tonality melody (Haydn) to light colors. This tendency is independent from the instrument. 4. DISCUSSION It is important to remember that subjects participating in the experiment were not told anything about the emotions, they rated the performances using the colors that best fit the particular expressive performance. The association of brighter colors to positive emotions and of darker colors to negative ones has been found in previous studies. This was also confirmed in an experiment in which young children were asked to associate colors to emotions [8]. Results from the present study, i.e. the association of brighter colors to the Haydn melody and I I I 0) g m LD m EE Figure 1. Color palettes resulting from the mean rating of the performances with expressive intention "shame"

Page  00000004 of darker colors to the Brahms melody, confirm the classical correspondence of positive emotions to major tonality and negative emotions to minor tonality [6]. The new contribution is that this association was obtained using colors instead of word lables or icons. One important parameter in the identification of emotion in music performance is the Sound Pressure Level (SPL) at frequencies above 1000 Hz. Since each melody is about 30 seconds long and the subjects are exposed to its different versions for on average 40 minutes, it appeared to be interesting to look at the long time average spectrum (LTAS) of the performances, it would probably show if there was a correlation between colors chosen by subjects and the SPL in the spectrum. The LTAS for each of the twelve performances of Haydn melody performed with saxophone is plotted in Figure 2. The angry and jealous performances have higher SPL in the spectrum at frequencies above 1000 Hz and the love performance has the lowest SPL above 1000 Hz. This corresponds with the preferred hues for these performances, i.e. red for angry and jealous, and blue and violet for love. Further investigations on the the relation between LTAS and color hue have to be done in the future in order to see if there is a consistent association between SPL and hue. If such a relationship were demonstrated, it would be interesting to rate perception of electroacoustic music using colors, since electroacoustic music usually explores new dimensions in the timbre space. Results from the present study show that subjects indeed used different color profiles for classifying the same piece of music, but these differences depended mainly on the performance (and on the instrument). This result could raise the question of whether differences reported in studies about color hearing could be explained in terms of differences in the expressive content of performances. The results from this work could be applied in the design of an application providing feedback about a performance. A possible application is a graphical interface showing acoustic cues extracted from expressive performances. Combinations of these cues correspond to a specific emotion intention that can be represented in terms of colors. In previous studies it has been demonstrated that auditory-visual interaction speeds stimuli discrimination capabilities [5] and that the surface color of an object affects its recognition [7]. Therefore, an interactive display in which expressivity in music is represented by shape, size, position, and color of objects on a screen (e.g. in a portable device) is expected to speed the process of learning and identification. For example, users could learn to associate red hue and square shapes to angry performances, and blue hue and round shapes to sad performances. A PD tool implementing this hypothesis has been designed and it will be tested in future usability tests. 5. ACKNOWLEDMENTS The present work has been supported by the previously mention Feel-Me project and by the EU funded S2S2 Co LTAS Hz Figure 2. Long Time Average Spectra (LTAS) of saxophone performances with 12 emotional intentions ordinating action, HUMAINE Network of Excellence, and ConGAS COST 287 Action. 6. REFERENCES [1] Cytowic RE (1989) Synesthesia: A Union of the Senses. Springer Verlag: New York. [2] Friberg A, Schoonderwaldt E, Juslin PN, Bresin R (2002) "Automatic real-time extraction of musical expression". Proc. of the International Computer Music Conference - ICMC 2002, G6teborg, 365-367 [3] Juslin PN (2001) "Communicating emotion in music performance: A review and a theoretical framework". in Juslin PN and Sloboda JA (eds)Music and emotion: Theory and research, OUP: New York, 305-333 [4] Juslin PN, Lindstr6m E (2004) "Musical expression of emotions: Modeling composed and performed features". Manuscript in preparation. [5] Marks LE (1987) "On cross-modal similarity: Auditory-visual interactions in speeded discrimination". J. ofExperimental Psychology: Human Perception and Performance, 13, 384-394 [6] Meyer LB (1956) Emotion and meaning in music. the University of Chicago Press: Chicago and London. [7] Tanaka J, Weiskopf D, Williams P (2001) "The role of color in high-level vision". Trends in Cognitive Sciences, 5:5, 211-215 [8] Zentner MR (2001) "Preferences for colours and colour-emotion combinations in early childhood". Developmental Science, 4:4, 389-398