Page  00000001 SONENVIR - A PROGRESS REPORT Alberto de Campo Christopher Frauenberger Institute for Electronic Music and Acoustics Robert Hoeldrich University for Music and Dramatic Arts Graz ABSTRACT This paper reports on an interdisciplinary research project on sonification called SonEnvir. We have begun working with five scientific disciplines in order to produce sonification prototypes that have practical and scientific value for these disciplines. These prototypes provide us with a body of approaches that will be integrated into a generalized sonification environment, and they serve as templates that are quickly adaptable to a wide range of other application fields. 1. OVERVIEW Starting from the research plan given in [3], the project has begun officially in January 2005, funded by the Styrian Future Fund. The center of the project is at IEM Graz (Institute for Electronic Music and Acoustics), and institutions from all four universities in Graz participate as target sciences: The Departments of Sociology and Theoretical Physics of Karl-Franzens-University, the Dept. of Neurology of the Medical University, the Dept. for Signal Processing and Speech Communication at Graz University of Technology, and the IEM (at University of Music and Dramatic Arts) itself with Acoustics research. In all of these target sciences, we have first results to present, all written on the same platform, which we expect will make for interesting demonstrations. 2. SOCIOLOGY After initial experiments with sociological data, e.g. election results predating the project, the first sonification prototypes made within the project are addressed to a very concrete application: In a European research project [8], a toilet for people with various disabilities is being developed; user interaction with this prototype has been recorded in log files, and sonification is currently being tested as one alternative evaluation/reconstruction method for these data. 2.1. Reconstructing Users' Actions For obvious reasons, visual recordings of the users are out of the question, thus it was decided to record logs of the user actions. In this test series, the toilet prototype was installed at a day care center for patients with multiple sclerosis (MS). Recorded data include: date and time, status of the log system, sensor data for state changes of the room (door open/closed), state of the toilet seat (height and tilt is measured), and a user interface in the room with buttons for height adjustment (up/down), tilt adjustment (up/down), flush, and alarm. The difficulty is to understand episodes of user interaction with this system; graphical display for so many dimensions has proven hard to read, and interesting information is expected to reside in the time structures, or patterns of usage. E.g. while there was user identification with RFID tags, these IDs were not always detected reliably, and classification of similar episodes (which may be attributed to specific users or groups of users) can be useful for more detailed evaluation of the practical usage of this prototype. 2.2. Sonification Design The initial sonification design for these log data is this: When the door opens, colored noise similar to diffuse ambient noise plays, and fades when the door is closed. Seat height and tilt are represented as continuous background drones, toggling between a reference pitch and a second pitch dependent on the distance from reference height/tilt. When changes occur (e.g. when a user sits down and thus affects height and tilt with his/her weight) the changing drone jumps to the foreground for a few seconds, then fades into the background again. User interaction with the buttons is straightforward to represent: When a button for height up was pressed, a glissando up is played, and a glissando down for the down button; tilt buttons use the same gesture with a different pitch, glissando rate, and timbre. The flush uses a noise burst, and the alarm button sounds like a door bell (this is mostly used to call for assistance, and only rarely for emergencies). To speed up learning time, a simple GUI displays the current state of the system graphically and as text; pressed buttons light up. P, L1UHI t uCM 1, L I Figure 1. FRR log display GUI. Playback of logs can be segmented into episodes automatically, and can be sped up (usually, one log is one entire calendar day), and various sound levels and tunings are user-adjustable. For more details, see [2]. 3. THEORETICAL PHYSICS The physics group works with models for quantum spectra of subatomic particles called baryons. The data under investigation are generated by different competing models that describe particle properties and symmetries.

Page  00000002 Baryons, most prominent among them the proton and the neutron, belong to the particle family of the hadrons. They are made up of quarks, the last known constituents of matter. The forces that determine their behavior are described within Quantum Chromodynamics (QCD). While QCD is solvable with arbitrary precision for high energies, there are still no satisfying solutions for middle and low energies. One must resort to effective theories or models, and one promising approach is relativistic constituent quark models. The aim is to set up models for baryons that allow for an effective description of their properties (like mass) and reactions based on QCD, and in accordance with all available experimental data for the low and middle energy regime. 3.1. Constituent Quark Models A typical CQM is based on a Hamilton operator, which consists of a kinetic energy term and a quark-quark interaction. The latter consists in turn of a confinement and a hyperfine interaction. Finding appropriate expressions for the interaction between quarks has been the focus of interest for decades. Quarks in nature are only found confined within hadrons; in QCD this is known as confinement. The form of the confinement potential depends on the model used, but generally it increases when one tries to separate two quarks. The hyperfine interaction was first modeled by a gluon exchange potential, which eventually failed to reproduce the mass spectra of light and strange baryons: One cannot model all excitation levels at the same time to fit experimental data. Similar problems occurred when modeling the hyperfine interaction with a superposition of gluon and meson exchange [7], and later on with the instanton-induced interaction. Some years ago the Graz research group suggested a hyperfine interaction based on the exchange of Goldstone bosons [10, 5]. This model is well suited to reproduce the ordering of energy levels by parity, better than the models given above. The models we chose for sonification exploration are the Goldstone-Boson Exchange model, the One-Gluon Exchange model, and for reference, a Confinement model which does not include a hyperfine interaction. 3.2. Baryon Classification Calculating baryon spectra is a typical few body problem, which is solved in terms of relativistic quantum mechanics. Here is a short explanation of the quantum numbers of baryons, which classify baryon states and determine its baryon wave function. When visualizing the masses and thus the excitation spectra of baryons, we use a classification scheme as in Fig. 2. A baryon is characterized by a specific name, which derives from the flavor F, a total momentum J, and the parity P. A baryon state is thus called e.g. N1/2+, meaning a nucleon with a total momentum of 1/2 and positive parity. There are six flavors of quarks: up, down, strange, charm, bottom and top. Up and down are very light, strange is a little heavier, and the others are very heavy. In Constituent Quark Models, mainly the baryons consisting of light and strange quarks have been investigated. The combination of three quarks with their flavors determines the name of each baryon type: The strange and light baryons can form nucleon N, delta A, lambda A, sigma I, xi E and omega Q particles. I.... Figure 2. Multiplet structure of the decuplet baryons as one example of baryon flavor symmetries. The lowest layer represents the sector of light and strange baryons. The total momentum or Spin of a baryon always is a multiple of 1/2, and the parity can be positive or negative. 3.3. Initial Sonification Questions While the basic properties of all these models can be read and interpreted from baryon spectra, there are a number of open research questions where we expect sonification to be helpful. We have started by identifying phenomena that are likely to be discernible in basic sonification experiments: Is it possible to distinguish e.g. the spectrum of an N1/2+ nucleon from, a delta D3/2+ by listening only? Is there a common family sound character for groups of particles, or for entire models? In the confinement model, the intentionally absent hyperfine interaction causes data points to merge into one: is this clearly audible? 3.4. Data Details Three specially made data files have been used so far, all of which contain mass spectra for nucleon and delta for one model each: file 1 is from a Goldstone-Boson exchange CQM, file 2 is from a One-Gluon exchange, and file 3 from a Confinement model. Each data file is made up of 20 lists, and each list refers to nucleon (or delta) for one value of J^P. The data sets are different lengths (22 - 2 entries), because we chose a mass limit for each data file. While the total number of data points is thus rather low, the interrelations and symmetries in the data are quite complex. In the current experiments, most of these symmetries are not portrayed yet. The most interesting dimension to start with is that of the mass differences, the level ordering. Because the energy level of the nucleon in its fundamental state is known to be 939 MeV in nature, one can shift all

Page  00000003 masses for each model accordingly; then the absolute values of the data (and the proportions between values) are also meaningful. 3.5. Initial Sonification Approaches Given the static nature of the data, and that the spacings between spectral lines are the main focus, a number of very simple strategies have been tried first: Mapping mass spectra to frequency spectra directly, with tunable transposition, and optional linear frequency shift and spreading Mapping (linear) mass spectra to a scalable pitch range, i.e. using perceptually linear pitch space as representation. In our current implementations both of these can be listened to as static spectra, as well as tunable arpeggios against a background drone of the same spectrum. Flexible comparison between different subsets of the data is a key requirement for static data. E.g. in order to do comparisons by parity, one can choose to play interleaved sequences alternating between parity + and -. These models are implemented in easily changeable SuperCollider3 scripts [9]; for more flexible browsing, a simple GUI has been made. All the tunable playback settings can be changed while playing, and saved for easy reproducibility and exchange of experimental settings. Some tuning options have been included to account for known data properties; e.g. to account for the diminishing reliability of the values calculated for higher excitation orders, we introduced a tunable slope factor in all models. Because this project has only started very recently, we do not have formal evaluations by domain experts yet; however, first informal tests show that there are audible family similarities across all particles in one model, and that the reduced complexity of the confinement model is well recognizable already in our first sonification designs. 3.6. Next Steps So far we have concentrated on sonifying mass spectra only; obviously introducing more particle properties will allow for richer representations that we expect to be of heuristic value. Apart from that, there is a number of strategies that we plan to explore: Comparing with experimental data, adjusting CQM parameters, using a model-based approach with macro-physical masses, using spatial ordering, and time phenomena. There are experimental data for the particle properties we deal with here. Because these have a known range of measurement imprecision, they cannot be compared 1:1 to the model data, but adaptations of the current synthesis models will allow to sonify these data. We expect comparing for family similarities between experimental and model data to be very interesting. We have produced data where the hyperfine interaction part of the quark-quark interaction is continuously 'turned up' from 0 to 100%. This data set may turn out to be interesting for understanding how the collapse of data points into single points proceeds, and for learning to follow and identify spectral changes like this by listening. Taking the notion of mass more literally and transferring it to the macro domain, we intend to put masses with the values from our data into mass-springdamping models with changeable, i.e. tunable nonlinear damping functions. While there is no directly plausible physical analogy for this kind of model (among other things, the quantum world in which baryons exist is relativistic), we expect that a coupled system of such masses will resonate in ways that express the family properties of a group of excitation spectra in a perceptually relevant fashion, because it will fit our everyday acoustic knowledge of complex resonators and their perceptual integration. As a variant, we also plan to explore a system of coupled pendulums (both in serial and parallel configurations). We have not explored more detailed spatial ordering by data dimensions yet, and we expect that navigation in a spatial order determined by symmetry relations between particle groups will be interesting. The simple option of spatial spreading of individual spectral line resonators already has turned out to be helpful for clarity of presentation. There are plenty of interesting time phenomena in the quantum domain which can be applied in numerous ways in further experimentation; e.g. there is enormous variation in the half life of the different particles, which could be expressed well in differentiated decay times for every spectral line. Finally, employing the probabilities for transitions between excitation states will allow for more dynamic models of the quantum world, and we expect that including particle behaviour and not just static properties will be quite intriguing. For more details, see [4]. 4. NEUROLOGY The neurology group mainly works with EEG data; this is a field where sonification has been used for quite a while, so there is a body of work to be analysed in order to understand the current state of the art. A number of possible approaches is given in [6], and one recent (and in our opinion very successful) example of extending the range of experimentation for both scientific and artistic purposes is [1]. EEG data are typically recorded in multiple channels of electrodes, usually less than 40, attached at precise spatial locations on the head, as defined in common standards (e.g. the 10-20 system. The interesting bandwidth of the electrode signals is ca. 0-50Hz, so typical sample rates are 256 Hz and higher. 4.1. Real-time Monitoring of Patients This is an application that will be immediately helpful in the EEG group's everyday work. E.g. patients with epilepsy are often monitored and recorded for several hours at a time, hoping to capture a seizure, which would allow for more precise diagnosis and hence more effective treatment. Both real-time EEG display and a video camera are being watched in order to react when a seizure happens; however, a number of things, e.g. actions like eye and body movements create artifacts that look silimar to

Page  00000004 seizures and thus may create false alarms, and sometimes the patients' appearance does not give any indication of a seizure occurring. This is where realtime audio monitoring could help to direct attention to a patient, and to classify what is happening - if the audio rendering is sensitive to these differences. 4.2. Current Development While this application is not nearly finished yet, we are making good progress toward it: We can read the standard data format, EDF files, into our platform of choice. For initial data screening we can use straightforward audification, speeded up by a changeable factor of ca. 10-100. Independent time-stretching and pitchshifting of the audified signal helps to focus on data regions of particular interest. Because some EEG recordings are long (up to 6 hours), we also need the option to access only particular regions that have been documented to be interesting, either while monitoring, or by visual inspection with the viewer software. The immediate next step is to implement filtering of the signals into the characteristic activity bands (alpha, beta, theta, delta, and slow cortical potentials), in order to be able to listen for different rhythmical phenomena in these different bands, and to sonify them separately. E.g. for some types of epileptic seizures, a very steady rhythm in the 1-2Hz region establishes itself. This looks very similar to normal activity in that band, because the more precise periodicity is hard to detect visually; we expect this phenomenon to be much easier to notice in an acoustic rendering. Finally, one can very easily render spatial information in the EEG signal, in order to know immediately which brain region shows activity that calls for more attention. 5. SIGNAL PROCESSING AND SPEECH COMMUNICATION A number of problems in Signal Processing are good candidates for sonification. One of the most interesting for us is the classification of noise signals: Whether a signal is purely stochastic or deterministic can not always be determined by statistical analysis, even though the methods for these classification problems have become extremely sophisticated. E.g. for initial data screening, audification is likely a good way to get a first impression, and to get some idea which statistical analyses to try first on an unknown signal. More sophisticated sonification approaches can of course employ pattern detection mechanisms built into human perception more subtly: We hope to succeed at making at least some deterministic qualities audible that can otherwise only be found with higher order statistics. Our current experiments are first steps toward this: We generate signals with well-understood distributions (e.g. uniform, gaussian, gaussian with moving average, poisson, etc.) and see how well they translate into perceptible qualities with different approaches, and whether they confirm the obvious expectations: E.g. a gaussian distribution used as a modulation source for pitch has a much clearer center pitch than a uniform distribution; mapped to the spatial positions of a cloud of grains, centeredness also comes out quite well. While this is not breaking any new ground yet, it is useful already as listening/training material: having practiced identifying typical distributions in textbook examples will be very helpful when similar distributions occur in real-world data from some other domain. 6. FUTURE WORK Considering the early stage of the project, we find we have made good progress with the results reported here. The immediate next step will be to study existing guidelines for psychoacoustically well-informed sound design strategies, test them, and apply them to our prototypes. Later on, we can start unifying the population of prototypes into a consistent body of easily adaptable examples for a larger range of possible applications; including ones approaching the transition zone between science and art. 7. REFERENCES [1] Barrass, S. et al, "Listening to the Mind Listening. A Concert of Sonifications at the Sydney Opera House." See http://www.icad.org/ websiteV2.O/Conferences/ICAD2004/concert.htm [2] Day6, C. et al, "Sonification as a Tool to Reconstruct Users' Actions in Unobservable Areas". submitted to International Conference for Auditory Display, Limerick, Ireland, 2005. [3] De Campo, A., Frauenberger, C., H61drich, R., "Designing a Generalized Sonification Environment", Proceedings of the International Computer Music Conference, Miami, USA, 2004. [4] De Campo, A. et al, "Sonification of Quantum Spectra". submitted to International Conference for Auditory Display, Limerick, Ireland, 2005. [5] Glantschnig, K., et al, "Extended Goldstoneboson-exchange Constituent Quark Model", Eur. Phys. J. A. (in print, 2005). [6] Hermann, T. et al, "Sonifications for EEG Data Analysis", Proceedings of the International Conference for Auditory Display, Kyoto, Japan, 2002. [7] L.Ya. Glozman, et al, "Unified Description of Light- and Strange-Baryon Spectra", Phys. Rev. D58, 094030 (1998). [8] Panek, P. et al., "Real Life Test with a Friendly Rest Room (FRR) Toilet Prototype in a Day Care Center in Vienna - An Interim Report". t.b.p. in Proc. 8th European Conference for the Advancement of Assistive Technologies in Europe, Lille, 2005. [9] http://sourceforge.net/projects/supercollider [10] Wagenbrunn, R. F., et al, "Covariant Nucleon Electromagnetic Form Factors from the Goldstone Boson Exchange Quark Model", Phys. Lett. B, 2001, v. 511, p. 33.