Page  1 ï~~COMPOSING WITH BRAINWAVES: MINIMAL TRIAL P300 RECOGNITION AS AN INDICATION OF SUBJECTIVE PREFERENCE FOR THE CONTROL OF A MUSICAL INSTRUMENT Dr M Grierson Goldsmiths Electronic Music Studios Dept. of Music Goldsmiths College ABSTRACT This paper presents a method for the stimulation and detection of P300 event related potentials in a real-time environment, and the application of this method for the sending of control signals to a computer music device. In the case of this study, an argument is advanced that the P300 can be triggered by certain types of visual stimuli, and that the following responses can be interpreted as an indication of subjective preference. Through this technique, a subject connected to an EEG can control a synthesiser or sequencer remotely without moving, by making a subjective decision to focus on a particular choice offered to them on a display. It is noted that this technique could be extended to a variety of contexts, not least the provision of a musical interface for the physically disabled. In addition, the implications of this study are discussed with respect to other work being carried out as part of the AHRC funded Cultural Processing project (Formerly C.A.V.E.). 1.INTRODUCTION "Perhaps within the next hundred years, science will perfect a process of thought transference from composer to listener. The composer will sit alone on the concert stage and merely think his idealized conception of his music. Instead of recordings of actual music sound, recordings will carry the brainwaves of the composer directly to the mind of the listener." Raymond Scott, 1949 [1] This quote is emblematic of many a composer's desire - to successfully communicate music directly from the mind to the minds of others, without need for the manual translation of musical ideas. Raymond Scott hoped that this would one day be possible with the aid of computers. There is as yet no method for translating music directly from mind to mind. However, there has been significant progress in the field of Brain Computer Interface (BCI) development that is beginning to make possible the translation of subjective states, however crudely interpreted, to information that can be employed in the real-time control of musical devices. This study presents one such method which has been demonstrated to work with a small group of subjects, and is being developed in the short term to allow for a usable level of musical interaction and analysis. 2.THEORY Event Related Potentials (ERPs) are brain signals that occur in response to external stimuli. They can be detected through the processing of an Electroencephalograph (EEG) signal. This method of brain signal measurement is effective for the identification of ERPs due to its relatively high temporal resolution when compared to other methods such as fMRI (functional Magnetic Resonance Imaging). As ERPs mainly occur in direct response to external stimuli, their identification is time sensitive to the onset of the stimuli. It seems logical to hypothesise that the relative temporality of ERPs can be seen as indicative of a particular type of neural processing, or - to be more specific, as the brain is a distributed processing network of circuit elements [2], the type of process occurring in response to stimuli can be said to be related to the amount of time taken for a neural signal to be processed by the network. Signals take time to be transferred from one part of the network to another, and the amount of time taken by extension may reveal the process taking place [3]. ERPs are at present seen as classifiable by these means, as they appear predictably across subjects given similar sensory stimulation within reasonable limits. Importantly, EEG has a relatively low spatial resolution, and although methods do exist for using EEG data to analyse the locations of specific brain activity, it is not the most effective method of doing so. ERPs are difficult to detect in real-time. They are normally revealed through the non-real-time analysis of a high number of EEG trials (20 or more). Commonly, a stimulus will be presented to a series of subjects, with each trial being defined by the time tagged onset of a stimulus. If an ERP is to be detected from the recorded signal, the EEG data must be heavily processed. ERPs manifest themselves as small variations in the signal occurring at a regular interval after the onset of the each stimulus. As the amplitude of an ERP is relatively low compared to the rest of the EEG signal, the signal needs to be averaged so that the non-random, time-dependent, event related signal can

Page  2 ï~~be measured. Non-relevant elements such as the subject's alpha waves and other background activity must be attenuated so that the ERP can be detected. Due to the sensitivity of EEG, any interference in the signal must be eliminated. Most importantly, trials which include high amplitude transients that peak above the resting state of the background signal must be discarded, as they are most likely caused by body movement. It is considered by some extremely difficult, if not impossible, to detect ERPs in a single trial due to the above considerations. However, despite this, there are several precedents [4], with some examples demonstrating a degree of success in detecting the P300 in single trials dating back to the 1960's [5]. These examples are considered non-contentious. It is for these reasons that this study chose to attempt a 'minimal' trial approach to EEG based ERP analysis - specifically, detection of the P300 (a relatively high amplitude ERP), with the hope of using its stimulation and detection as a real-time (yet still quite slow) control method. The P300 occurs approximately 300 milliseconds after the onset of a stimulus, and can be triggered by an 'oddball' event, such as a momentary deviation from a pattern. It is thought by some that the P300 is an indication that a subjects working memory is being updated, and as such is considered an unconscious response varying in latency depending on working memory capacity [6]. It was hypothesised that the presentation of various choices flashing at regular timed intervals would be a simple method suitable for the testing of new detection algorithms. This paradigm has been well tested in non-musical scenarios, and is considered robust (see below). 3.PREVIOUS WORK G.tec systems have been instrumental in the development of ERP based BCI research in a variety of disciplines for over a decade [7]. Their systems have featured in a number of significant experiments on real-time analysis of brain signals, including single trial P300 detection for navigating virtual environments [8]. Spencer et al have examined the issues of speed in the utilisation of the P300 for BCI [9]. These studies demonstrate that a "minimal" trial P300 based BCI is feasible. However, speed issues need to be overcome by more effective stimulus delivery and signal processing approaches. Other methods for real-time non-invasive BCI are currently being demonstrated by commercial companies such as Emotiv [10] and Neurosky [11 ]. These devices employ real-time signal processing techniques alongside dry electrode based hardware, and are being aimed at the general public. At present, neither of these devices have specifically musical applications being developed for them. In addition, it is not thought that these devices use ERP detection as the primary method of operation. To date, BCI research for music has centred on the analysis of spontaneous potentials and sensory motor signals [12]. David Rosenboom [13] details a number of early experiments that deal with biofeedback approaches. In addition, spontaneous potentials are the basis of Alvin Lucier's 1965 piece, Music for Solo Performer. A rich and informative historical survey of the development of EEG and brain-related musical composition approaches is detailed by Miranda & Brouse [14]. Miranda et al comment on the difficulty of interpreting meaning from an EEG signal, and describe a musical BCI system based on biofeedback principles where users train themselves to respond to their own spontaneous potentials. Analysis of spontaneous potentials is very robust. However, meaning is difficult to extract using this particular method. All of the above methods for Music BCI use spontaneous potentials and biofeedback. The approach described here differs in that it employs ERPs, where both the cause and the function of the EEG signal - the ERP - is considered to be understood. Therefore, meaning can be derived from the EEG signal by encoding the context within the system itself in the form of an ERP-evoking stimulus, given certain caveats widely considered best practice [15]. As it is the derivation of meaning, however slow and cumbersome, which is of specific interest to this project, it has been adopted as the method of choice. 4.METHOD The method described here was designed as a proof of concept, and is offered as such. It is a modification of a standard paradigm for real-time ERP detection which is known to be reliable. An improved version of this method is being more fully developed with a larger group of subjects, for which results are still outstanding. This research was conducted with the use of a Guger Technologies (g.tec) 8 channel biosignal amplifier (funding for which was provided jointly by both the Research Office and Music Department of Goldsmiths College). The g.tec device is entirely suitable for Brain Computer Interface work, including minimal trial ERP detection. G.tec provides a Simulink blockset for operation in the Matlab environment. This blockset includes a set of basic signal processing blocks, and an elementary P300 test environment. The performance of Matlab for the purposes of real-time musical interaction was judged by this researcher to be somewhat unreliable. In addition, the presentation and control of customisable OpenGL accelerated highspeed visual stimuli was not judged effective enough in the Matlab environment. This is significant, as it is the visual presentation which stimulates the P300 component. Furthermore, even in cases where simple real-time musical interaction and basic visual stimulation was possible, there were concerns that it would interfere with the real-time EEG analysis required to reliably enable the system's functionality. For these reasons it was decided to split the processing load onto two machines. One for performing initial processing on the EEG signal, and a second for displaying the stimulus, sending possible ERP time-tags and producing pitched signals. In order to achieve this, it was necessary to find a communication method between

Page  3 ï~~Matlab 2007b and Max/MSP/Jitter 4.6.3 over a network. Initial problems were encountered when attempting to use Open Sound Control, as Matlab OSC functionality is broken at the time of writing. However, it was possible to use Simulink network send and receive blocks to send data to and from Max/MSP/Jitter and Matlab. In this way, 16 bit 64hz signals can be sent as byte arrays to Max/MSP over the network, and then decoded. In the method described here, only the controlling data is passed between each machine, not the EEG signal itself, although current tests have been successful in streaming live EEG signals to Max/MSP, and a driver is under development. The test was performed on five subjects, being only a pilot study, and given that the detection method is considered non-controversial. Electrodes were positioned using the international 10/20 system [16]. A single electrode was placed at position Cz (top), and the ground on the forehead. In addition, a reference signal was taken from the right mastoid (behind the ear). These areas were prepared with abrasive gel, and electrode gel was applied in order to lower the resistance to appropriate levels. Each subject was placed approximately 2 feet in front of a display. The screen displayed up to five octaves of note-names, labelled according to the MIDI specification. The display would flash each note-name a set number of times in a pseudo-random order. Each notename would flash red for approximately 50 milliseconds and then remain dark for a random period up to 1800 ms. The Subject was asked to stare at a chosen note-name, and mentally count each time the note-name flashed red. This was considered a suitable minimal P300 ERP component generation technique and complies with the experimental method employed in other successful real-time P300 experiments including those detailed above. It is also coherent with the approach proposed by g.tec for P300 ERP component generation. The EEG signal was processed as follows. First, the signal was low-pass filtered to remove any data above 30hz. Each time a note-name flashed up on the screen, a 500 millisecond chunk of EEG data was saved to a buffer. Buffers were stored with reference to the appropriate note name, and averaged together at the end of a set number of trials. If the amplitude peaked significantly above the average maximum amplitude of the signal, it was assumed that the specific trial had been contaminated by non-EEG data, such as physical movement, and discarded automatically. At the end of each run, the system collated a set of averaged EEG data, time-tagged to particular notenames, and checked for errors. The signal would then be scanned for peaks. If the averaged signal contained an amplitude peak within a reasonable range, and the peak occurred between 200 and 500 milliseconds after the onset of the stimulus, it was judged to be a possible P300 component. The most favourable result would be the peak with the highest amplitude. The machine would then respond by playing the note associated with the detected P300, and then store the note in a file for playback at a later time as part of a sequence. The number of trials was progressively lowered each time a run was completed. It was established that a C major scale from C3 to C4 would be a suitable test pattern for checking the system's functionality. Each subject attempted to pick out the 8 note scale in the correct order, one at a time, out of the given 36 note-name choices. The large range of note-name choices has the affect of slowing the system down considerably. However, it was decided that the large range of choices would significantly reduce the odds of the correct notename choice being selected accidentally, therefore being a more effective test of the system. 5.RESULTS A control experiment was initiated employing a pseudorandom 16bit 256hz signal in place of the EEG signal. The system failed to perform the C major scale, or any elements of it in the correct order over a period of 10 complete test runs. However, it did at times erroneously detect notes from the scale, although the detection rate was not above chance. Four of the five subjects were able to operate the P300 BCI with a 75% or above success rate (6 out of 8 notes successfully detected in sequence) with seven trials. With thirty six possible note-name choices, at a rate of 50ms per flash, the BCI took just over twelve seconds to attempt to decode the subjective choice made by the subject. One subject was able to play almost all of the C major scale using the device with as few as four trials per note, reducing the detection time to seven seconds. It is acknowledged that this is slow when compared to some other BCI devices, even those that use the P300. However, the subjects involved had no training prior to the session, and the number of choices was beyond that which is strictly necessary. Each individual note-choice has a one in thirty-six chance of being correct. In many BCI environments, the number of possible choices is significantly reduced, lowering the time taken for each trial. It was decided that, for the means of testing ERP component detection for musical interaction, a large number of choices would increase the validity of the study, making it coherent when set against P300 spelling devices, and providing a reasonable test bed for improving both ERP stimulation and signal processing methods. A major obstacle to the BCI's operation may have been discarded trial data as a result of unintentional body movement. The detection rate is severely hampered by discarded trials, as the averaging process is dependent on multiple passes to operate effectively. In addition, the sound being generated may have had a negative effect on the detection process, although this test provides no data on this. The data suggests that the subject who failed to gain a 75 % success rate at seven trials had a large number of trials discarded due to amplitude peaks over the average amplitude of the EEG background noise - suggesting head movement, body movement and/or electrical interference. It would not be an expected result if the paradigm did not function at all, as the paradigm itself is not in question. The only question that this study seeks to address is purely one of feasibility for information retrieval in a musical environment.

Page  4 ï~~6. CONCLUSION In the light of these results, it was decided that minimal trial ERP methods were both possible and appropriate for the development of Brain Computer Music Interfaces and related research. This is for two reasons. First, the system gives significantly better results than spontaneous potential and biofeedback methods where large numbers of choices are required, and where subjects have had no training. Second, as the system can be effective with only 7 note name choices, this reduces the time to just over two seconds. With a faster flash rate, and a smaller number of choices, the system becomes even more responsive. 7.FUTURE WORK This study forms part of the AHRC funded Cultural Processing project (formerly C.A.V.E.). This project aims to explore cognitive and structural approaches to contemporary computer aided audiovisual composition. This test establishes that ERP methods can be employed so that a user's subjective choices can be unconsciously probed through visual means, provided that the user is paying attention to the stimuli. The above-chance detection of P300 ERPs with untrained subjects on this initial test is good grounds for continued research in the area. A program of ERP-based real-time experimentation has been initiated using both visual and sonic material. These activities will explore other ERP components with respect to audiovisual composition, perception and cognition, whilst the Music BCI project continues as a test bed for the study of ERP classification. It is hoped that this project will aid in the development of new technologies for audiovisual interaction, including a Brain Computer Music Interface for the physically disabled. However, it is the possibilities for information retrieval within an arts research environment, and not exclusively BCI development, which continues to drive this research. 8.REFERENCES [1], Last accessed 08/02/2008 [2] Nunez, P.L. & Srinivasan, R., Electric Fields of the Brain The Neurophysics of EEG, Oxford University Press, 2006 [3] Otten, L. J. & Rugg, M. D., "Interpreting EventRelated Brain Potentials", in Handy, T.C., (ed.), Event Related Potentials. A Methods Handbook. MIT Press, 2005 [4] Spencer, K M, "Averaging, Detection, and Classification of Single Trial ERPs" in Handy, T C (ed.), Event Related Potentials, a Methods Handbook. MIT Press, 2005 [5] Woody, C.D., "Characterization of an adaptive filter for the analysis of variable latency neuroelectric signals". Medical and biological engineering, 5, 539-553, 1967 [6] Nittoni, H., Nageishi, Y., Nakajima, Y., & Ullsperger, P., "Event-related potential correlates of individual differences in working memory capacity", Psychophysiology 36: 745-754 Cambridge University Press, 1999 [7] Polikoff, J.B., Bunnell, H.T., & Borkowski Jr., W.J., "Toward a P300-based computer interface", Proceedings of the RESNA '95 Annual Conference, RESNAPRESS, Arlington Va.,1995 [8] Bayliss, J.D., & Ballard, D.H., "Single Trial P300 Recognition in a Virtual Environment," Computational Intelligence.: Methods & Applications Proceedings, Soft Computing in Biomedicine, 1999. [9] Spencer, K.M., Karni, O., Wijesinghe, R., & Donchin, E,. "The mental prosthesis: assessing the speed of a P300-based brain-computer interface," BCI- Technology: Theory and Practice, First International Meeting, Rensselaerville, 1999 [10] Last accessed 08/02/2008 [11] htt:// Last accessed 08/02/2008 [12] Miranda, E. R., Roberts, S. & Stokes, M. "On Generating EEG for Controlling Musical Systems", Biomedizinische Technik, 2004 [13] Rosenboom, D. Biofeedback and the Arts.: Results of Early Experiments, Vancouver: Aesthetic Research Centre of Canada, 1975 [14] Miranda, E., & Brouse, A., "Toward Direct Brain-Computer Musical Interfaces", Proceedings of the 2005 International Conference on New Interfaces for Musical Expression, Vancouver, BC, Canada, 2005 [15] Otten, L. J. & Rugg, M. D., "Interpreting EventRelated Brain Potentials", in Handy, T.C., (ed.), Event Related Potentials.: A Methods Handbook. MIT Press, 2005 [16] system (EEG Last Accessed 01/06/08