Page  00000001 PLYMOUTH BRAIN-COMPUTER MUSIC INTERFACE PROJECT: INTELLIGENT ASSISTIVE TECHNOLOGY FOR MUSIC-MAKING Eduardo Reck Miranda, Andrew Brouse, Bram Boskamp, Hilary Mullaney Computer Music Research, School of Computing, Communications and Electronics University of Plymouth, Drake Circus, Plymouth PL4 8AA, UK http://cmr.soc.plymouth.ac.uk ABSTRACT This paper introduces a system that uses brainwaves, or EEG (electroencephalogram), information to compose and play music in real-time. The system composes music using generative grammars and transition rules controlled by means of information extracted from the EEG of the subject. The paper starts by noting the various attempts at the design of systems that make music from the EEG signals, followed by a short technical introduction to the EEG. The generative component of the system is inspired by the work of David Cope [7] on computer-replication of a musical style by analysis of given musical examples. The paper concludes with a brief discussion on the contribution of our research for the development of assistive technology for physical disability. 1. INTRODUCTION Human brainwaves were first measured in 1924 by Hans Berger [3]. Today, the EEG has become one of the most useful tools in the diagnosis of epilepsy and other neurological disorders. In the early 1970s, Jacques Vidal did the first tentative work towards a system to communicate with a computer with the EEG. The results of this work were published in 1973 in a paper entitled Toward Direct Brain-Computer Communication [21]. This field of research is known as Brain-Computer Interface (BCI) and there is a growing number of researchers worldwide working in this field. Many attempts followed with various degrees of success. In 1990, Jonathan Wolpaw and colleagues developed a system to allow primitive control of a computer cursor by subjects with severe motor deficits [22]. For recent reports on BCI research please refer to the special issue of IEEE Transactions on Biomedical Engineering published in June 2004 (Vol. 51). We are devoted to the development of BCI systems for musical applications (Brain-Computer Musical Interfaces or BCMI) and we pay special attention to the development of generative music techniques tailored for such systems. We are very interested in the possibility of using BCMI as assistive technology to enable people with severe physical disabilities to have the opportunity to make music. As early as 1934, a paper in the journal Brain had reported a method to listen to the EEG [1]. It is now generally accepted that it was composer Alvin Lucier, who composed the first musical piece using EEG in 1965: Music for Solo Performer [13]. Pioneers such as Richard Teitelbaum [20], David Rosenboom [19] and a few others followed with a number of interesting systems and pieces. Back in 1975 David Rosenboom edited a remarkable book on the topic [18] and more recently Andrew Brouse published a comprehensive review on using brainwaves to produce music [6]. Our research builds on the work developed by these pioneers in a number of ways. Firstly, we are employing and developing more sophisticated analysis techniques to harness the EEG signal. Furthermore, we are developing new psychophysical experiments in order to gain a better understanding of the EEG components associated with musical cognition and methods to train subjects to generate such EEG components. Finally, we are developing generative techniques especially designed for musical composition and performance with a BCMI. This paper focuses on the first and the latter. More information on our psychophysical experiments can be found in [14, 15]. Figure 1. Demonstration of the BCMI-Piano using a Disklavier piano. Before we proceed, note that the BCI research community understands that a BCI system is a system that allows for the control of a machine by explicitly thinking the task(s) in question; e.g., control a robotic arm by thinking explicitly about moving an arm. This is an extremely difficult problem. The system presented in this paper does not address this type of explicit control. This would be even more difficult in the case of music. However, we are not interested in a system that plays a melody by thinking the melody itself. Rather, we are furnishing our systems with Artificial Intelligence in order to allow them make their own interpretation of the meaning of the EEG patterns. Such machineinterpretations may not always be accurate or realistic, but this is exactly the type of man-machine interaction that we are addressing in our work. 2. THE ELECTROENCEPHALOGRAM (EEG) Neural activity generates electric fields that can be recorded with electrodes attached on the scalp (Figure 2): the electroencephalogram, or EEG. These electric fields are extremely faint, with amplitudes in the order of only a few microvolts. In order to be displayed and/or

Page  00000002 processed, these signals must be greatly amplified [16]. The EEG is measured as the voltage difference between two or more electrodes on the surface of the scalp, one of which is taken as a reference. The EEG expresses the overall activity of millions of neurons in the brain in terms of charge movement, but the electrodes can detect this only in the most superficial regions of the cerebral cortex. The EEG is a difficult signal to handle because it is filtered by the meninges (the membranes that separate the cortex from the skull), the skull and the scalp before it reaches the electrodes. Furthermore, the signals arriving at the electrodes are sums of signals arising from many possible sources, including artifacts like the heartbeat and eye blinks. Figure 2. Brainwaves can be detected with electrodes place on the scalp. There are a number of approaches to quantitative EEG analysis, such as power spectrum, spectral centroid, Hjorth, event-related potential (ERP) and correlation, to cite but five. A brief non-mathematical introduction to EEG power spectrum and Hjorth analyses is given below due to their relevance to the systems introduced in this paper. A discussion on other analysis techniques and how they have been used in neuroscience of music research can be found in [4, 10, 11], to cite but three. Power spectrum analysis: is derived from techniques of Fourier analysis, such as the Discrete Fourier Transform (DFT). This is useful because the distribution of power in the spectrum of the EEG can reflect certain states of mind. For example, a spectrum with salient lowfrequency components can be associated with a state of drowsiness, whereas a spectrum with salient highfrequency components could be associated with a state of alertness. There are five recognised frequency bands of EEG activity, also referred to as EEG rhythms, each of which is associated with specific mental states: delta, theta, alpha, low beta and high beta rhythms. There is, however, some controversy as to the exact frequency boundaries of these bands and the mental states with which they are associated. Hjorth analysis: is an interesting time-based method [9], which measures three attributes of the EEG: its activity, mobility and complexity. This method represents each time step (or window) using only these three attributes and this is done without conventional frequency domain description. The signal is measured for successive epochs (or windows) of one to several seconds. Two of the attributes are obtained from the first and second time derivatives of the amplitude fluctuations in the signal. The first derivative is the rate of change of the signal's amplitude. At peaks and troughs the first derivative is zero. At other points it will be positive or negative depending on whether the amplitude is increasing or decreasing with time. The steeper the slope of the wave, the greater will be the amplitude of the first derivative. The second derivative is determined by taking the first derivative of the first derivative of the signal. Peaks and troughs in the first derivative, which correspond to points of greatest slope in the original signal, result in zero amplitude in the second derivative, and so forth. Activity is the variance of the amplitude fluctuations in the epoch. Mobility is calculated by taking the square root of the variance of the first derivative divided by the variance of the primary signal. Complexity is the ratio of the mobility of the first derivative of the signal to the mobility of the signal itself. A sine wave has a complexity equal to 1. There is no clear agreement as to what these measurements mean in terms of mental states. It is common sense to assume that the longer a subject remains focused on a specific mental task, the more stable is the signal, and therefore the lower is the variance of the amplitude fluctuation. However, this point questions the possible affects of fatigue, habituation and boredom, which we have not yet accounted for in our research. 3. THE BCMI-PIANO SYSTEM The BCMI-Piano (Figure 1) falls into the category of BCI computer-oriented systems. These systems rely on the capacity of the users to learn to control specific aspects of their EEG, affording them the ability to exert some control over events in their environments. Examples have been shown where subjects learn how to steer their EEG to select letters for writing words on the computer screen [5]. However, the motivation for the BCMI-Piano departed from a slightly different angle from other BCI systems. We aimed for a system that would make music by guessing the meaning of the EEG of the subject rather than a system for explicit control of music by the subject. Learning to steer the system by means of biofeedback would be possible, but we did not investigate this possibility systematically yet. We acknowledge that the notion of "guessing the meaning of the EEG" here is simplistic, but it is nevertheless plausible: it is based on the assumption that physiological information can be associated with specific mental activities [2]. The system is programmed to look for information in the EEG signal and match the findings with assigned generative musical processes corresponding to different musical styles. The BCI-Piano is composed of 2 main modules: analysis and music engine. The EEG is sensed with 7 pairs of gold EEG electrodes on the scalp, roughly forming a circle around the head. A discussion for the rationale of this configuration falls outside the scope of this paper. It

Page  00000003 suffices to say that we are not looking for signals emanating from specific cortical sites; rather, the idea is to sense the EEG over the whole surface of the cortex. The electrodes are plugged into a biosignal amplifier and a real-time acquisition system. The analysis module: performs power spectrum and Hjorth analyses in real-time. The analysis module generates two streams of control parameters: one stream contains information about the most prominent frequency band in the signal and is used by the music engine to generate the music. In the current version, the system activates rules for two different styles of music, depending on whether the EEG indicates salient lowfrequency or high-frequency components (or EEG rhythms). The other stream contains information about the complexity of the signal and is used by the music engine to control the tempo of the music (Figure 3). analysis music engine EEG IVMIDI Figure 3. Spectral information is used to activate generative music rules to compose music on the fly and the signal complexity is used to control the tempo of the music. The music engine module: is a set of generative music rules, each of which produce a musical bar, or measure. Basically, the system works as follows: every time it has to produce a bar, it checks the power spectrum of the EEG at that moment and activates rules associated with the most prominent EEG rhythm in the signal. It can generate music that contains, for example, more Schumann-like elements when the spectrum of the subject's EEG contains salient low-frequency components and more modem or jazzy elements when the subject the spectrum of EEG contains salient highfrequency components. Example-based musicalgeneration systems are often based on formalisms such as transition networks or Markov Chains to re-create the transition-logic of what-follows-what, either at the level of notes [12] or at the level of similar "vertical slices" of music [7, 8]. For example, David Cope uses such example-based musical-generation methods but adds phrase-structure rules, higher-level composition structure rules, and well-placed signatures, earmarks and unifications [7]. The act of recombining the building blocks of music material together with some typical patterns and structural methods has proved to have great musical potential. We have chosen to stick to a statistical predictor at the level of short vertical slices of music such as a bar or half-bar, where the predictive characteristics are determined by the chord (harmonic set of pitches, or pitch-class) and by the first melodic note following the melodic notes in those vertical slices of music. We added a simple method of generating short musical phrases with a beginning and an end that also allows for the real-time influence from a given EEG-signal. The system generates musical sequences by defining top-level structures of sentences and methods of generating similarity- or contrast-relationships between phrases. We are currently exploring other possibilities with extra constraints and transformations on music that will generate music with repeated similarities (such as David Cope's "unifications" [7]), and larger structures such as the generation of rondo's and variations, while still adhering in real-time to the demands of the EEG-signal. Figure 4 shows an example generated by our system with elements from the musical style of Robert Schumann and Ludwig van Beethoven. The former corresponds to alpha rhythms and the latter to beta rhythms; the user sets these associations arbitrarily beforehand. In this example the EEG would have jumped back and forth from bar to bar between the two EEG rhythms. The harmonic and melodic distances are quite large from bar to bar, but still they are the optimal choices in the set of chosen elements from the two composers. beethioven Shmn &~4~ emnn b bethve n Figure 4. An example of a generated mixture of Robert Schumann and Ludwig van Beethoven. The system is initialised with a reference tempo (e.g., 120 beats per minute), which is constantly modulated by the signal complexity analysis (i.e., Hjorth analysis). The music engine sends out MID! information for performance; we implemented a demonstration using the Disklavier piano, manufactured by Yamaha. 5. CONCLUSION: BCMI AS ASSISTIVE TECHNOLOGY AND FUTURE RESEARCH With this research we hope to open up many possibilities, as both a recreational device for people with disabilities and as an instrument for concert performance and composition. At present, access music tutors use gesture devices and adapted accessible technology to make this possible, which achieve excellent results with people with learning and physical disabilities. Although for people with severe physical disabilities, having complete control of the environment

Page  00000004 created for them by the facilitator can sometimes prove difficult. For many with disabilities, EEG signals could be the only option of control and sometimes with others be a more reliable one, due to the nature of their disability. To have greater control over this system, we are developing methods to train subjects to achieve specific EEG patterns to play the BCMI-Piano system. We have initial evidence that this can be made possible using a technique known as biofeedback. Biofeedback technology is used to treat and control a number of conditions; examples include migraine headaches and epilepsy. In addition it has been used for artistic expression through music, performance and visual art [18, 19]. As yet, there has been little research into the area of training people with disabilities to control BCMI systems. The aim of our current research in this area is to create a methodology to train subjects to use the technology through the use of biofeedback. We acknowledge that there still remain a number of cumbersome problems to be resolved before we can realise our ultimate goal: an affordable, flexible and practically feasible BCMI. One of the key issues needing to be addressed is the problem of interpreting the meaning of the EEG. Although powerful mathematical tools for analysing the EEG already exist, we still lack a good understanding of their analytical semantics in relation to musical cognition. However, continual progress in the field of cognitive neuroscience [17] is improving this scenario substantially. Another aspect that needs to be developed is the non-ergonomic nature of the electrode technology for sensing the EEG. REFERENCES [1] Adrian, E.D., and Matthews, B.H.C. (1934), "The Berger Rhythm: Potential Changes from the Occipital Lobes in Man." Brain, 57(4): 355-85. [2] Anderson, C. and Sijercic, Z. (1996), "Classification of EEG signals from four subjects during five mental tasks", Solving Engineering Problems with Neural Networks: Proceedings of the Conference on Engineering Applications in Neural Networks (EANN'96), pp. 507-414. [3] Berger, H. (1969), "On the Electroencephalogram of Man", The Fourteen Original Reports on the Human Electroencephalogram, Electroencephalography and Clinical Neurophysiology, Supplement No. 28. Amsterdam: Elsevier. [4] Besson, M. and Faita, F. (1995), "An event-related potential (ERO) study of musical expectancy: Comparison of musicians with non-musicians", Journal ofExperimental Psychology: Human Perception and Performance 21:1278-1296. [5] Birbaumer, N., Ghanayin, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kubler, A., Perelmeouter, J., Taub, E. and Flor, H. (1999), "A spelling device for the paralysed", Nature 398:297 -298. [6] Brouse, A. 2004, "Petit guide de la musique des ondes c6r6brales", Horizon 0, No. 15. http://www.horizonzero.ca, Accessed 31 January 2005. [7] Cope, D. 2001. Virtual Music. Cambridge, MA: The MIT Press. [8] Cope, D. 1996. Experiments in Musical Intelligence. Madison, WI: A-R Editions Inc. [9] Hjorth, B. (1970), "EEG analysis based on time series properties", Electroencephalography and Clinical Neurophysiology 29:306-310. [10] Janata, P. and Petsche, H. (1993), "Spectral analysis of the EEG as a tool for evaluating expectancy violations of musical contexts", Musical Perception 10(3):281-304. [11] Koelsch, S., Schroger, E., Gunter, T.C. (2002), "Music matters: Preattentive musicality of the human brain", Psychophysiology 39(2002): 38-48. [12] Kohonen, T., Laine P., Tiits, K and Torkkola, K. (1991), "A Nonheuristic Automatic Composing Method", P. Todd and D. G. Loy (Eds.), Music and Connectionism. Cambridge (MA): The MIT Press. [13] Lucier, A. (1976), "Statement On: Music for Solo Performer", D. Rosenboom (Ed.), Biofeedback and the Arts, Results of Early Experiments. Vancouver: Aesthetic Research Center of Canada Publications. [15] Miranda, E.R., Sharman, K., Kilborn, K. and Duncan, A. (2003), "On Harnessing the Electroencephalogram for the Musical Braincap", Computer Music Journal 27(2):80-102. [16] Miranda, E. R., Roberts, S. and Stokes, M. (2004). "On Generating EEG for Controlling Musical Systems", Biomedizinische Technik 49(1):75-76. [19] Niedermeyer, E. and Lopes da Silva, F. H. (Eds.) (1987) Electroencephalography (2nd edition). Munich: Urban and Schwartzenberg. [20] Peretz, I. and Zatorre, R.J. (2003), The Cognitive Neuroscience of Music. Oxford: Oxford University Press. [22] Rosenboom, D. (Ed.) (1975), Biofeedback and the Arts, Results of Early Experiments. Vancouver: Aesthetic Research Center of Canada Publications. [23] Rosenboom, D. (1990), Extended Musical Interface with the Human Nervous System, Leonardo Monograph Series No. 1. Berkeley, California: International Society for the Arts, Science and Technology. [25] Teitelbaum, R. (1976), "In Tune: Some Early Experiments in Biofeedback Music (1966-1974)", D. Rosenboom (Ed.), Biofeedback and the Arts, Results of Early Experiments. Vancouver: Aesthetic Research Center of Canada Publications. [27] Vidal, J.J. (1973), "Toward Direct Brain-Computer Communication", L. J. Mullins (Ed.), Annual Review of Biophysics and Bioengineering, pp. 157-80. [28] Wolpaw, J., McFarland, D., Neat, G., Forneris. C. (1991), "An Eeg-Based Brain-Computer Interface for Cursor Control", Electroencephalography and Clinical Neurophysiology 78(3):252-9.