Page  121 ï~~MELONET: Neural Networks that Learn HarmonyBased Melodic Variations Johannes Feulner and Dominik Hornel Institut fur Logik, Komplexitat und Deduktionssysteme, Universitat Karlsruhe, D-76128 Karlsruhe, Germany johannes@ira. uka. de Abstract MELONET, a system that can harmonize melodies and do melodic variations that are well bound to harmonic contexts, is presented. MELONET comprises several neural networks that work on various subtasks. The topology reflects the fact that music is a phenomenon that occurs simultaneously at several time scales through neurones firing at different frequencies. This allows melodies to be learned more naturally since they can be treated as sequences of (multi note) motifs instead of as sequences of single pitches. In order to let the motifs as they evolve during the learning process (based on real music) correlate to musical intuition a representation scheme is proposed that codes pitches in a distributed fashion relative to their harmonic context. 1 Introduction There are several approaches to modelling melodies using neural networks. Most notably Todd [1991], Mozer [1991] and Freisleben [1992] proposed models that are being based on the assumption that melodies are sequences of pitches. However, many melodies are composed out of little melodic cells i.e. motifs. If a neural network is to learn such melodies, its architecture should reflect the type of melody under consideration. Even where Freisleben goes on to model two part pieces of music, he considers them as two simultaneously occurring pitch sequences, making it difficult to capture the harmonic properties inherent in any kind of tonal music. MELONET demonstrates that neural models can favourably model melodies as sequences of harmony based motifs. Combining the harmonic modelling of HARMONET [Hild et al 92], [Feulner 93] together with the SYSTEMA [Hornel 93] approach to motifs leads to a model that overcomes the aforementioned shortcomings. 2 Task Description MELONET was built in order to learn melodic variation from examples found in the literature. As test cases the chorale variations of J. Pachelbel I I I I I K I A il I VI W, Ad X11 W, FW I I I 1 I Figure 1 Partita super "Christus, der ist mein Leben" by Johann Pachelbel a (top): Beginning of the four part chorale b (bottom) Melodic variation of soprano voice ICMC Proceedings 1994 121 Neural Nets

Page  122 ï~~(1653-1706) were used. One such chorale variation (also known as a partita) is based on a one part melody. It consists of several movements. The first movement is always a four part chorale. The following movements are then constructed as melodic variations of one or more of the voices of the four part chorale (cf. Fig 1) Thus the task that MELONET performs is achieved in two steps. First a plain melody is given as an input to the HARMONET based chorale harmonizer. Then, in a second stage, one of the chorale's voices is varied. To learn the proper harmonization and variation techniques the networks comprising MELONET receive from other neurons at time t multiplied by the weight associated with it over several time steps T. Then this sum is passed through the sigmoid activation function (here the logistic function) to produce a new activation output. t si(t) f( '(sj(k)w1 -.-Oi)) fort mod T-O s1(t) - f (O) otherwise. These neurons integrate their input signals over a certain time span, thus their output reflects long term information about the melody input. Adapting the RPROP algorithm [Braun and Riedmiller, 1992], an extremely fast variant of the backpropa Input: MELODY, plan % % I % MNZ,...... output: melodic Figure Topology of MELOI a " MELONET ' variation 2 SETs networks gation algorithm, to handle delayed update neurons is fairly straight forward. The weights leading to TDU neurons are kept constant between update cycles. The weight updates as computed by RPROP are memorized, summed, and only added to the actual weights whenever t mod T = 0. 4 Topology The topology of MELONET is similar in spirit to the proposal of Todd [Todd 91] for the layout of a hierarchy of networks operating at different time scales. As figure 2 shows, a window sliding over the melody produces input into the HARMONET assembly of networks. The output of were trained on the data of the original Pachelbel chorale variations. The extraction of features and the learning process itself is automated. There is no manual labelling of training sample involved. The details of the first step will not be discussed in this paper. See [Feulner 93] for a detailed account. 3 Delayed update neurons One of the networks of MELONET uses a special kind of neurons, delayed update neurons (T-DU neurons). Delayed update neurons do not fire each time their input changes but rather at discrete time intervals. They sum up the activations si(t) they Neural Nets 122 ICMC Proceedings 1994

Page  123 ï~~HARMONET is fed into the supernet, a feedforward network with delayed update neurons. Since all variations of Pachelbel do have sixteenth notes and the melody inputs quarters as smallest rhythmical values, T was set to 4. This means that the weights of the subnet change four times more often than the weights of the supernet. This allows the interpretation of the supernet output as a "plan" in the sense of Todd steering the behaviour of the subnet. Based on the plan provided by the supernet which is constant for one melody quarter (T = 4) the subnet will fill out such a quarter with sixteenth notes. The plans correspond to certain types of motifs the subnet has to produce. There is feedback from the subnet output to subnet and supernet input. This feedback it fed into some of the input neurons to give MELONET a long term memory. This is the same approach as that used in HARMONET. 5 Representation - octave - tenuto, if p prolongs same pitch This coding is distributed with respect to pitch. There is no absolute pitch reference. A pitch is being looked at as embedded into a harmonic context. This is one of the decisive points in order to get a musically sensible generalization behaviour out of MELONET's networks. 6 Performance MELONET is capable of producing melodic variation in the style of J. Pachelbel. The training examples were automatically extracted form Pachelbel's compositions. The combination of supernet/subnet was trained as one network. Thus there was no explicit requirement on what plans would be the output of the supernet. This makes it possible to adapt MELONET to different kinds of music. The importance of the plan units was revealed in an experiment where the number of plan units was changed. Figure 3 shows that with 5 plan units the style of variation is mostly linear T T a r melOdv pitch p is consonant dissonant p refers to next/prey, note direction (up/down) harmonic distance octave tenuto -0 1 0 1 - 1 - 1 11 - - 1 - --0 -U 0 0 0 Figure 2: One of the most important aspects of virtually any neural network application is a proper representation of the input data. In order to produce the right plans for motifs to be implemented by the subnet. Of special importance is the harmonic output representation devised for the subnet's output (cf. SYSTHEMA [Hornel, 1993]). This representation codes pitches as depending on certain harmonic contexts (cf. Figure 2). The features coding a pitch p include: - p is consonant with respect to its harmony - if p is dissonant does it refer to its successor or to its predecessor - direction of p (up, down to next pitch) - if p is consonant: distance to base note 1 1 1 0 1 1 1 1 1 1 0 1 - 1 - -1 - - - - - 1 - 1 1 -1 1 1 1 1.1 1 1 - 1 - 1 - 2 2 1 1 1 - 0 - 0 - 0 0 0 0 0 - 0 0 0 0 OO0 0 0O The representation whereas with 10 plan units arpeggios prevail. During the composition process it is possible to gradually activate the plan units. This way it is possible to obtain musically valid interpolations that are bound to a certain musical style. In the example given (Figure 4) it is possible to smoothly change between linear progressions and arpeggios. Even though the style of melodic variation of MELONET is easily distinguished from Pachelbel's its output is musically valid. The strong points are the consideration of the harmonic context yielding a proper dissonance handling and the variations produced are not arbitrary pitch sequences but concatenations of musical motifs. ICMC Proceedings 1994 123 Neural Nets

Page  124 ï~~7 Conclusions Modelling melodies as sequences of musical motifs rather than as sequences of pitches gives a musically relevant generalization when new, unforeseen melodies are to be composed. Employing a representation scheme that codes pitches relative to a harmonic context facilitates the construction of motifs that are bound to specific harmonic situations. In order get a neural model to automatically recognize and compose in terms of motifs the use of delayed update neurons in modules dealing with long term aspects of the data are a good means. The approach of having a supernet that learns the melody as a sequence of motifs and a subnet to trans Computer Music Conference, ICMA Tokyo 1993 [Freisleben, 1992] B. Freisleben. The Neural Composer: A Network for Musical Applications. Artificial Neural Networks, no 2: pp.1663-1666, Elsevier 1992 [Hild et al., 1992] Hermann Hild, Johannes Feulner, Wolfram Menzel. HARMONET: A Neural Net for Harmonizing Chorales in the Style of J.S. Bach. In Advances in Neural Information Processing 4 (NIPS 4), R.P. Lippmann, J.E: Moody, D.S. Touretzky (eds.), pp.267-274, Kaufmann 1992 [Hormel, 1993] D. Hornel SYSTHEMA - Analysis and Automatic Synthesis of classical Themes. Proc. of the 1993 International Computer Music e - Â~ Â~ IW1 Figure.4 Two variations composed by MELONET a: 5 plan units, linear style b: 10 plan units arpeggio style late the motifs into single pitches looks promising for further exploration of models with modules working at many different time scales. References [Braun and Riedmiller, 1992] Heinrich Braun Martin Riedmiller.RPROP: A Fast Adaptive Learning Algorithm. In International Symposium on Computer and Information Sciences VII 1992, Proceedings. E. Gelenbe, U. Halici, N. Yalabik (eds.), pp. 279-286, EHEI Press, Paris 1992 [Feulner, 1993] J. Feulner. Neural Networks that Learn and Reproduce Various Styles of Harmonization. Proc. of the 1993 International Conference, ICMA Tokyo 1993 [Mozer, 1991] Michael C. Mozer. Connectionist Music Composition Based on Melodic, Stylistic, and Psychophysical Constraints. In Music and Connectionism. P. Todd and G. Loy (eds.), pp.195-211, MIT Press 1991 [Todd, 19911 Peter M. Todd. A Connectionist Approach to Algorithmic Composition. In Music and Connectionism. P. Todd and G. Loy (eds.), pp.173-194, MIT Press 1991 Neural Nets 124 ICMC Proceedings 1994