Page  444 ï~~An Artificial Neural Net for Harmonizing Melodies Dan Gang and Daniel Lehmann Institute of Computer Science Hebrew University Jerusalem 91904, Israel fax: 972-2-658-5439 e-mail: {dang, lehmann}Â~cs.huji.ac.il Abstract Whenever a musician needs to harmonize or analyze harmonization of a given melody, after establishing the key, he/she tries to extract, for each measure, some harmonic hint from the melody in the form of key notes. These key notes provide a sparse harmonic outline that will be completed to a full harmony using constraints derived from: the harmonic and the melodic contexts, the musical style and culture and the individual musical taste. Based on this analysis, we propose a Jordan's sequential neural net the structure of which reflects the way the human musician proceeds: it contains a sub-net that identifies key notes. The net is capable to learn simple harmonized melodies and generalizing what it has learned by harmonizing melodies it has never seen. 1 Introduction Suppose you have heard today a hit on the radio, you can sing it with the singer and you almost know all the words. You would.like to play it to your friends at a party in the evening. You take the guitar and try to find the right chords, but the result is not convincing. It is really important to you, so you meet with your guitar teacher who can probably help you to find the chords. You sing the song and it takes only two minutes to your teacher to find the guitar accompaniment! "Tell me how you do it", you ask your teacher. But the teacher says: "I think I can not give you the algorithm or the exact rules for doing it, and moreover it depends on my mood, and it took me some years before I learned to do it well, so keep trying!". Harmonizing a melody is one of many musical activities that is difficult to explain or teach. A lot of people get frustrated when they are unable to find sensible chords for a melody. This paper will try to explain how musicians solve the problem of harmonizing a simple melody and will present a hierarchical neural network that effectively learns this task. We have tested the ability of an artificial neural network to learn the task of harmonizing very simple songs. The class of the songs is Western European, and the melodies and harmonies are at the level of a child after two years of playing the organ. There are two reasons for this choice of melodies. First, we want to understand the basic chord functions and how they are learned by beginners. Then, the aesthetic judgment on harmonizing simple melodies is probably less controversial. 2 The Problem What gives an expert the ability to harmonize a melody so easily and so well? What makes a child after two years of playing the organ, able to cope with the harmonization problem? We do not claim 444 I C MC P ROC EED I N G S 1995

Page  445 ï~~we understand these processes fully, but we shall describe the steps a musician may use, and based on these steps, the architecture of the system that is capable of learning how to perform the task. 2.1 The Human Musician's Approach Whenever a musician needs to analyze the harmonization of a given melody or to create an harmonization for a melody, he/she examines the notes of the melody in order to establish the key or the tonal centers. Then, using the knowledge that the harmonic rhythm is generally of the order of magnitude of a measure, he/she considers the notes in every measure, in order to extract some harmonic hint: key notes that comprise a sparse harmonic outline which will be filled and completed using constraints derived from: the harmonic and the melodic contexts, the regularities of the chord progressions in the musical style and culture, and the individual musical taste. Not all experts agree on how one finds the key notes, or on their exact relation with the chord finally chosen. 2.2 The Computational Approach A neural net for harmonizing simple songs has been built and tested. It is capable of learning harmonized melodies, using a back propagation algorithm, and generalizing what it has learned by harmonizing melodies it has never seen. We avoid the need to locate the key of the song by considering only songs written in Do major. In section 2.1 we described how a musician tries to extract a harmonic hint for a measure, by notes in every measure. In our system, the task of finding key notes is delegated to a sub-net that is trained separately. These key notes are a function of the notes that appear in the measure only: no melodic context is taken into account. The full harmonization is produced by a Jordan's sequential net [Jor86], described in Section 3. This net learns the regularity of the chord progressions in the musical style from examples. The harmonic context is used directly in this net, together with the key notes, to create a full harmonization, i.e., the chords. The melodic context is used only indirectly by the net, through the key notes produced by the sub-net. 3 Description of the network The architecture of the network is described in figure 1. The sub-net for key notes learns to extract lelodyN ar'omlinq Figure 1: The net architecture ICMC PROCEEDINGS 1995 445 445

Page  446 ï~~key notes out of the melody. It learns from examples: harmonized melodies. To define the key notes for a measure, in our examples, we used the intersection between the notes appearing in the melody and the notes that build the chords for the measure. The sub-net is a 3-layer standard feed forward network trained by back propagation. The input layer represents the notes of one measure of the melody and the output layer represents the key notes with twelve units for the chromatic scale. The 4-layer sequential net learns the sequence of chords, with consideration to the key notes that are propagated from the sub-net (See Figure 1). The output units represent 14 chords - 7 for the Do Major scale's degrees, and 7 more for their Dominant 7 chords. The output layer is fed back in the state units of the input layer for the computation of the next chord: the value of a state unit at time t is the sum of its value at time t - 1 multiplied by some decay parameter and the value of the corresponding output unit at time t - 1. 1 The state units represent the context of the current chord sequence, and the output layer represents the prediction of the net for the next chord. The net also includes two internal hidden layers: Hidden and HiddenDis. The HiddenDis layer represents the chromatic scale: each of the twelve units represents a pitch. This layer is only partially connected to the output layer (for example: units Do, Mi, Sol are connected to the C unit in the output layer, 2 but unit Do is not connected to the G unit). The output layer of the key notes sub-net is fully connected to the HiddenDis layer, but some of the connections are fixed and some are learnable. For example: Do in the key notes sub-net output layer excites Do in HiddenDis layer, and therefore, indirectly, excites the C unit of the output layer; it also inhibits Si in HiddenDis and, indirectly, inhibits G in the output layer. In this way, we impose soft constraints on the way information is represented in the HiddenDis layer. The plan units in the input layer, are used to label different sets of notes' sequences. They have been used to harmonize in different styles, but this experiment will not be described here. To implement the system architecture we used the PlaNet software [Miy91]. A full description of the network and the results obtained may be found in [Gan94]. 4 Running the Net 4.1 The Learning Phase For the learning phase, the net was given eighteen simple melodies with chords, and learned to reproduce these examples. We have tested the performance of the network with several different learning parameters (e.g., decay, number of hidden units). The net was able to learn the task relatively fast. 3 4.2 The Generalization Phase The network's generalization capability has been tested by giving it new melodies to harmonize. The results were compared to the original harmonization of the source. A trained musician has evaluated the resulting chords from the aesthetic and functional aspects. In the generalization process the net was tested by giving it four new melodies, and evaluating the-resulting chords. These four new melodies were gradually growing in complexity, with regard to the density of the harmonic rhythm and the use of chord substitution. The resulting harmonization was found to be functionally appropriate by trained musicians. For some songs, the result obtained was almost identical with the harmonization proposed by the source, a music study book for children. ' For the other songs, the chords obtained were surprisingly aesthetic, flexible and different from the original. ' These feedback connections do not appear in the figure of the net architecture. ' The bottom left side of the figure, that is denoted by kn, is describing only the output layer of the sub-net for key notes. 3 For a decay parameter = 0.3, 5 hidden units and 18 songs it took 50 epochs to learn to reproduce the songs with a very small error. 4Se the first song (the left side), "Lane in the Forest" in the Appendix. The upper Line of the chords describes the chords resulting from the net. 446 IC M C PROCEED I N G S 1995

Page  447 ï~~The chords resulting from the harmonization for the song "Supercalifragilisticexpialidocios" have been functionally appropriate, 5 mainly because, the major functions in measures 4, 8, 12 and 16 are identical with the harmonization proposed by the source. The net, nevertheless, lacks consideration of the traditional musical hierarchy. This is revealed by the fact that the rhythm of harmonic changes is not symmetric. In measure 15 the G7 chord that gives us the sense of ending is missing, because the net lacks sense of the melody structure. This is to be expected, due to the rules of the gamewe have imposed on the net - it has information about the melody, only one measure ahead. The chords resulting from the harmonization for the song "Soleado" 6 are functionally appropriate, surprisingly aesthetic, flexible and different from the original. This song is the most complex in the set of the generalization learning examples, because of the density of the harmonic rhythm and the use of some chords substitutions. 5 Summary and Acknowledgments We achieved our goal of building a neural net for harmonizing melodies, by dividing the task into two sub-tasks. After achieving the first sub-task (key notes), we try to achieve the second subtask (harmonization), under the influence of the first. This neural network approach uses a simple representation and returns impressive results. It, nevertheless, lacks consideration of the traditional musical hierarchy. There is evidence that the harmonization may be extended to include individual and different musical tastes and styles. In [Tod89], Todd suggested the use of hierarchical sequential networks. This work is evidence that such networks, the structure of which reflects musical cognitive experience, are very promising. At the methodological level, probably the most important lesson of this work is, that one may use musical knowledge to direct the choice of the structure of the system. Acknowledgments: the help of Naftali Wagner in preparing this paper is gratefully acknowledged. This work was partially supported by the Jean and Helene Alfassa fund for research in Al. A The Net's Harmonization for New Melodies. Lane in Lb. IFor t e.......-._.--..I. c GC7 c C G t C C/G G7 C C - Vo)C C)C CA C. p. C C C C C C C C 7 C C C ' C V C7E Am C7 C cC C C C 0 *N) C C C C " C7 La Am C7 C References [Gan94] Dan Gang. A neural network that learns to harmonize melodies. Master's thesis, Institute of Computer Science, Hebrew University, 91904 Jerusalem, Israel, September 1994. [Jor86] M.I. Jordan. Attractor dynamics and parallelism in a connectionist sequential machine. In Proceedings of The Eighth Annual Conference of the Cognitive Science Society, Hillsdale, N.J., 1986. [Miy9l] Yoshiro Miyata. A User's Guide to PlaNet Version 5.6. Computer Science Dept., University of Colorado, Boulder, 1991. [Tod89] P.M. Todd. A connectionist approach to algorithmic composition. Computer Music Journal, 13:27-43, 1989. SSee the second song (in the middle), in the Appendix. 6See the third song (the right side), in the Appendix. ICMC PROCEEDINGS 199547 447