Page  1 ï~~TECHNOLOGY, CULTURE AND LOCATION IN A RECENT NEW ZEALAND SONIC ART PROJECT Ian Whalley University of Waikato Department of Music ABSTRACT The HIEMPA (Hybrid Instruments from Electroacoustic Manipulation And Models of Pitorino and Aquascape) project combined a team of people with technical, artistic, environmental and cultural expertise toward the artistic outcome of extending the New Zealand sonic art tradition. The project involved collecting audio samples from the aquascape of the Ruakuri Caves and Nature Reserve in Waitomo, South Waikato, New Zealand; and samples of a variety of Pitorino - a New Zealand Maiori wind instrument. Following a machine learning analysis of this audio material and an analysis of the performance material, hybrid digital instruments were built and mapped to suitable hardware triggers. The new instruments are playable in real-time, along with the electroacoustic manipulation of Pitorino performances. The project takes into account the environmental and cultural significance of the source material, with the results to be released as a set of compositions. This paper discusses the research process. 1. INTRODUCTION AND STARTING POINTS The HIEMPA project team based in the Waikato, New Zealand, commenced with the aim of further exploring/developing the national sonic art tradition. The project required technical, cultural, environmental, compositional and traditional Maiori instrumental knowledge. Philosophically, we began from attempting to combine and extend two different New Zealand sound art traditions: environmentally based electroacoustic music, and traditional Maiori instrumental music. The New Zealand electroacoustic music composition tradition began from the pioneering work of Douglas Lilburn (1915-2001). Lilburn often looked to environmental sources as the basis for his works in an attempt to find a New Zealand voice [3]. Subsequent generations of New Zealand composers continued this tradition [7], later demonstrated in some of the recent works on the New Zealand Sonic Art CD Series. A characteristic of the approach, particularly with the use of digital technology, is using editing/construction methods to create works based on audio sample playback, rather than using technology to play and manipulate samples in real-time as a type of instrument. Further, despite a performance based 'soundculture' using found and invented instruments in New Zealand by composers such as Phil Dadson [7], 'instruments' from the natural environment have infrequently been digitally extended in this type of work. The artistic initiators of the HIEMPA project then sought to extend the New Zealand electroacoustic music tradition by using different methods of real-time sound generation for environmental sounds, and also extend the performancebased environmental 'found instrument' tradition by adding real-time digital sound manipulation. The broader sonic art tradition in New Zealand has recently seen a revival of traditional Maiori instruments, due to the late Hirini Melbourne, Richard Nunns, and instrument maker Brian Flintoff. Traditional instruments are played live on by Melborne/Nunns on their recent albums [1], and particularly in later work, compositions are only possible with the use of multi-tracking to build layers of sound in the studio with the aid of the producer [8]. Further, to get collections of instruments to blend beyond what players may improvise, instrument volumes and EQs have been extensively manipulated, with instruments that are nearly naturally silent appearing strongly in the final mix. In addition, the manipulation of spatial and pan effects is an integral part of the structure of many of the compositions on the CDs, creating a new hybrid style that is as much a reflection of contemporary technology as it is traditional instruments [8]. A limitation is that performers cannot recreate the works in real-time. Further, the timbre, small dynamic range, and small pitch range of most of the melodic instruments is restricted to their original capabilities on the CDs. Additionally, performers do not manipulate effects in real-time; the producer, as part composer of the works, added spatial/pan effects later. The HIEMPA team intended to extend this pioneering work by digitally modelling one family of traditional instruments, the Pitorino, to be able to play beyond its current pitch limitation; and also extend its real-time performance capabilities by applying digital sound effects beyond spatial/ pan effects processing to both an acoustic and a physically-modelled Pitorino that the performers could manipulate in real-time. Technically, the HIEMPA team also had some starting focal points. We had access to expertise and extensive tools in machine learning and data mining, an area of international research at Waikato University in the Computer Science. Although this technology had been applied to some aspects of traditional European instrumental performance in research worldwide apart from Waikato University, it had not been used to analyse Maori instruments' timbres or performance values: useful for new digital instrument building. Digital instrument building is widespread commercially, but a main limitation with many commercial/ 'pop' instruments is the types of synthesis (wavetable, K-S) generally, which does a poor job at replicating melodic instruments in particular. Further, keyboards are a poor means of expressively triggering digital models of most wind instruments [5], and many effects options added to commercial synthesisers are largely generic rather than designed as part of an instrument's performance parameters. The team proposed to make a collection of purpose built digital instruments based on the traditional wind instrumental source material collected, using appropriate real-time

Page  2 ï~~hardware triggers, generated through physical modelling synthesis. From a technical perspective, there was also a need to develop a means of digitally controlling and manipulating samples of environmental sounds using MAX/MSP; and to find an interesting way to digitally extend audio samples of struck stones that we intended to collect as 'found instruments'. Finally, there was a need to put all the 'instruments' envisaged for the ensemble into a playable configuration: to be able to create 'hyper' instruments based on exploring new gestures and sonic combinations from different combinations of the ensemble. An aesthetic approach and context was also needed, to give the resulting work an authentic and located voice. In response, we first sought to embed the work in our local environmental and cultural surroundings. This was to be tempered and later realised in through musical composition drawing on the team's artistic strengths in New Zealand electroacoustic music and traditional instrumental music making. 2. CORE TEAM AND ROLES All core team members had a connection to the Waikato area. Buddy Te Whare represented the Maiori people whose tribal area we entered to gather the samples. He also later provided the working title for the project in collaboration with the composer. Caver John Ash, who has a long association with the Waitomo area in the South Waikato as an expert guide, arranged clearance for all sites we needed to record at for the project. He also initially led the team into the Ruakuri cave/reserve. Richard Nunns supplied original samples of traditional Maori instruments for analysis, and suggested some electroacoustic manipulation of them to provide ideas toward building the hybrid instruments, and played a number of musical sequences at two locations in the cave that we recorded to check the characteristics of the reverberation space. Bernhard Pfahringer from the Computer Science Department at Waikato University undertook the data analysis of the audio samples; applying techniques to extract useful patterns on which to base instrument models. Digital Music Studio Director at the University of Waikato, Ian Whalley acted as the project director. He aided with collecting all sound material, and subsequent audio analysis. He built all digital instruments, assembled the software controller interfaces for the instruments, selected appropriate realtime triggers for them, and designed/configured the hardware rig for the composition process. He programmed the MAX/MSP patches for the control and manipulation of the ensemble, and is the composer who will create the compositions from the research work. 3. LOCATION, DATA, COLLECTION The Waitomo area of the South Waikato of New Zealand is internationally regarded for its extensive collection of ancient limestone caves. Our team's focus was on one of the lesser-known cave systems called Ruakuri, and the associated Ruakuri nature reserve. We selected this newly reopened cave because it had a greater range of sonic possibilities and fewer visitors than the main Waitomo Cave. A one hundred meter vertical steel spiral staircase reaches to the cave floor and the walking path through the cave is 1.6 kilometres The cave is carved out by a combination of erosion and water pressure over thousands of years. Embedded in the walls are ancient fossils of large sea creatures, from a time when the underground terrain was part of the sea floor. From a cultural perspective, local legend suggest that a young Maiori, hunting for birds, discovered Ruakuri four to five hundred years ago. As wild dogs inhabited the entrance of the cave, it was named Rua (den) kuri (dogs). We made several trips to the area to get appropriate water samples, as the quality of this work depended on the amount of rain that had fallen prior to each visit. Subsequently, we first spent many hours collecting samples of surface water from the streams in the Ruakuri Reserve. From the riverbed, we also recorded samples of resonant stones being struck together. Most of our subsequent audio recording work took place underground, gathering water samples from within the Ruakuri cave system. A main concern in the underground excursions was to get samples that were different from each other, and unique to the Ruakuri Cave. It often took a number of recordings to get isolated samples from the many locations in the cave that we found interesting sonically. Further, although the sound of water is not unique internationally, a combination of factors at different locations in the cave gave each sample unique characteristics that reflected the environment. We came to think of the cave a type of meta-instrument, where each space with dripping and running water contributed a unique combination of timbral, rhythm, reverberation, and ambience. From this, a set of samples was built to reflect the total sonic environment. By chance when we were leaving the cave on one of our trips, there were a group of Tui, a New Zealand native bird singing in the trees at the entrance to the cave. The timbral range of the birds is extensive, and provided an unexpected set of samples that were to be later useful in reflecting the cultural context of the composition framework for the project, particularly given the story of how the caves were discovered. The second area of data collection involved sampling the Piltorino, an instrument that Richard Nunns choose for the project partly because of its potential to fit with water samples sonically; and his understanding that it would be culturally appropriate as an instrument to use. Recordings included single notes, and a series of performance gestures played on the instruments by Richard Nunns. In addition, Richard recorded some traditional percussion instruments in response to the rhythm of the water samples that we collected. 4. DATA ANALYSIS & INITIAL EXPERIMENTATION After the samples were edited, Ian Whalley and Bernhard Pfaringer began analysing the audio samples using various techniques to extract essential data. The main tool for this, along with others, was the Data Mining & Machine Learning Suite (WEKA) from Waikato University. The focus was primarily on the Piltorino samples and Tui (native bird) samples, which we

Page  3 ï~~anticipated attempting to model and later extend through physical modelling synthesis, to make new melodic instruments. In setting out, we expected to be able to extract dynamic timbral data and performance data (i.e. gestures) from the samples. While able to track some of the performance data using machine methods, capturing all parameters proved more difficult. Through extensive trials however, we found a method of extracting the performance and dynamic timbre data reasonably accurately, at least for the traditional wind instruments. Performance data, such as the level and shape of vibrato, glissandos and attack characteristics, was quicker and more accurate to replicate on an approximated physical model of the Pitorino by acute listening/musicianship, and trial and error tweaking parameters of the physical modelling software. Machine analysis provided an accurate sense of dynamic timbre as the instruments' responded to various performance gestures. We then mapped the performance and timbral data responses into a physical modelling synthesiser module in preparation for finding suitable hardware triggers for these. Parts of the bird sample analysis were also completed using this method in the first instance, but the timbral range of Tui proved too difficult to model. We therefore decided to treat most of the bird samples as environmental instruments; the same way we were to approach the water samples. We also undertook an analysis of the reverberation of samples that we collected in the vertical shaft at the entrance of the cave, and one of the large spaces at the back of the cave, based on a sample of Richard Nunns playing a small traditional nose flute in the spaces, envisaging that the reverberation characteristics may be useful in the composition process. Finally, we analysed audio aspects of the rock samples that were played at the site, by which we hoped to make alternative consonant scales based on Sethares' work [4] on the relationship between tuning, timbre, spectrum and scales. His method shows the connection between the structure of a scale and the structure of the sound used, illustrating how musical consonance or dissonance does not depend on set intervals, but on aligning tuning with the spectrum and timbre of the sound. This allows composers to make consonant music using tunings apart from twelve-tone equal temperament. Some initial instrument experimentation was then undertaken. In testing our electronic version of the Pitorino it was found to be a useful extension played away from its original three-note range. In tandem with this, we set up a series of tests routing the acoustic Pitorino through various effects units, to try and build up taxonomy of useful compositional aesthetic possibilities that could later be quickly called on. The traditional percussion instrument samples were left in their original state, after deciding only to apply pan and reverberation effects after being triggered by a sampler. We did this because they were not part of the Ruakuri environment, but could be used to add a sonic counterpoint and sense of propulsion in the composition process. The water and bird samples were loaded into the sampler, and taxonomy developed of compositional possibilities developed based on changing a variety of sample parameters. Once again, the results were fed through a range of effects units to build taxonomy of compositional possibilities. To finish, we experimented with a number of scales away from equal temperament that might be used for the rock sound models, after adjusting spectra and timbre to make them consonant in each scale we thought might be useful. 5. INSTRUMENTS, RIG, TRIGGERS After deciding on a suitable physical modelling synthesis package and loading in the timbral shaping and performance data, the sonic output of the models of the Piltorino were checked against the source material to a point of musical satisfaction with what we were able to achieve. Not unexpectedly, our instrument never matched the acoustic instrument's subtlety in the hands of a skilled player in its original pitch range, but our electroacoustic extension of it was musically interesting. Stones Birds dater; Surface and,.Colle cae. samp Traditional wind 8 percussion P ercussion' Footboard trigger ~fing er trigg er~ N' Electro ni 11performani WJind triggers syinthesizerf..................................................................................................................................................................................................................................................................... (our=e Building real-time performance Rig and routing instruments paths ', sampler manipulatin FoortEboard: yn F' MAX/MSS _Cr p _. t he Culture......... itlo. En....o..... '_ ire Ero.,irunrcient Figure 1. Project overview A performance rig was then designed to be played by a maximum of two people: one playing the original acoustic Piltorino, and one manipulating electronic aspects of the inputs. The unit included a computer connected to a number audio processing modules, largely using the computer for control and physical modelling synthesis, and other units for audio processing/generation. The selection and combination of these units was one of aesthetic choice by the composer. To control the units, we began by testing and subsequently selecting suitable real-time performance triggers/controllers, aiming to make each 'instrument'

Page  4 ï~~playable in real-time as expressively as possible. This began by making a multi-switch footboard with expression pedals available to the acoustic Pitorino player, whereby they could trigger effects changes in realtime while playing into a microphone. The physical model of the Pitorino was attached to a wind synthesizer, also mapped to be able to change/control effects in real-time. Models of the stone samples' timbre and spectra were made in a synthesiser, as it proved easier to control tuning this way with a MAX/MSP patch. To trigger the water, birds and traditional percussion samples and control associated effects in real-time percussion and finger controllers were used together with an additional footboard controller. To allow configuration changes to be made in real time, patches were developed in MAX/MSP. Various software based interfaces for these triggers were tested, and a package suitable for programming and mapping the output of the controllers to instrument module inputs selected. A hardware rig was then assembled that wired together the control aspects of the instruments through MIDI, digital audio routing to effects, and audio out mixing to a stereo output. Finally, in combining all instrumental patches and effects combination discovered through tests, a master patch library was constructed that could be navigated and controlled in real-time through MIDI manipulation. 6. INSTRUMENT AND ENSEMBLE TESTING The Pitorino models created were then electroacoustically extended through trials, based mainly on extending a taxonomy of performance gestures from the original acoustic recordings to keep the gestural connection to the live instrument. The set of effects extended in the virtual instrument tests, allowing the modelled version to develop its own characteristics. To allow the audio stuck stone models to be quickly played in scales in more than one key centre, Sethares Adaptun MAX/MSP patch was adopted [4]. The rig setup, like all ensembles, was then considered as one instrument, a hybrid of acoustic and electroacoustic inputs. Trials and reprogramming were then carried out combining aspects of the sample input, live input, electronic instrument input and effects processing to discover what combination worked together most successfully, and what felt the most musically playable in real-time. The individual instrument sounds were then added to a meta-patch library. In addition, the meta-library included interesting sounding combinations of inputs discovered in the experimentation process. Developing taxonomy to be able to understand the best groupings of hybrid instrument sounds is an ongoing process. Some initial tests were carried out to see what sort of composition one might improvise in real-time using a small section from the library. In doing this, it became apparent that it was going to take some time to master playing the ensemble in real-time, due to having to manipulate a variety of inputs simultaneously. To cut down the complexity of performance control, some small generative patches were created in MAX/MAP to trigger combinations of sounds using processes that were suited to the theme of the piece, a technical process that the composer had previously used with combinations of traditional instruments [8], allowing performers to react to the sonic contributions that the machines provided. Consideration was also given to adding intelligent agent software to allow the ensemble to be more interactive with performers [6] but this remains to be explored. 7. CULTURE, COMPOSITIONAL AND CONCLUSION From this work, Ian Whalley and Buddy Te Whare discussed a cultural and environmental basis to tie together a collection of compositions. In arriving at a suitable title, Nga Pahupahu o Ruakuri (the voices or speaking of Ruakuri), we attempted to link traditional Maiori views of the Ruakuri area with an historical and contemporary sense of the environmental dimensions of the project. Using a narrative approach, artistic work would then explore the link between past and present, the idea of descending from the world of light to darkness, stations within the cave, and the return to the present. With the background research largely completed, focus has now shifted to the composition process. This will involve a combination of preconceived sonic structures, themes and narrative shapes; real-time performance/improvisation, and MAX/MSP patch triggering to create much of the content. Balancing innovation and creativity [2], we now have the tools to extend Lilburn's notion of a national voice and sonically explore further aspects of what it means to live in contemporary New Zealand. 8. REFERENCES [1] Melbourne, H. Nunns, R. CD Te Hekenga-airangi. Rattle. RAT DV010, 2003; CD Te Ku Te Whe. Rattle. RAT D004, 1996. [2] Milicevic, M. "Deconstructing Musical Structure" Organised Sound, 3(1):27-34, 1998. [3] Norris, M., Young, J. 2001. "Half Heard Sounds in the Summer Air: Electroacoustic Music in Wellington and the South Island of New Zealand". Organised Sound 6(1): 121-28, 2001. [4] Sethares, W. Tuning, Timbre, Spectrum, Scale (Second Edition). London: Springer, 2005. [5] Whalley, I. "Real-time to hyper-real gestures through physical modelling synthesis". 10th International IASPM Conference, UTS, Sydney, Australia, 9-13 July: 395-401, 2000. [6] Whalley, I. "PIWeCS: Enhancing human/machine agency in an interactive composition system". Organised Sound 9(2): 170-176, 2004 [7] Whalley, I. "Recent New Zealand Electroacoustic Music: Nexus, Flux Negotiation". Leonardo Electronic Almanac. v12(8): 1-8, 2004. [8] Whalley, I. "Traditional New Zealand Maiori instruments, composition and digital technology." Organised Sound 10(1): 57-65, 2005.