Page  272 ï~~A graphical environment for electroacoustic music composition David Rossiter and David M Howard Parallel and Signal Processing Applications Research Group, Department of Electronics, University of York, Heslington, York, Y01 5DD, ENGLAND Electronic mail: { dpr,dh} Abstract Interfaces used by composers of electroacoustic music tend to constrict the creative process because of their inappropriate design. This paper describes the design and implementation of a system developed for electroacoustic music composition in which constrictions of compositional development are greatly diminished. The user is abstracted away from low-level transformations by visual representations of the sound elements and the processes they undergo. The composer is able to configure vision and sound elements as the piece develops. Because of this, a maximum cognitive connection is made between the aural imagery within the mind of the composer and the visual imagery used to represent the sound on the screen. The graphical area in which these icons are arranged is therefore both an interface to the creative process and a notationally true score of the composition. Keywords: electroacoustic music composition, notation, score, graphical user interface (GUI) 1 Introduction In recent years, the use of computers to create music has spread dramatically, affecting nearly all areas of composition and performance. Computers have led to the growth of new modes of musical and artistic expression which might not otherwise have developed. As computer music has developed, experimentation by composers building upon the potential of electronically produced sound has revealed the limitations of the traditional musical representation, the notated score. This has frustrated attempts to notate new musical ideas and expression. New compositional and sound synthesis techniques in addition to developments in music performance have highlighted the need for a different style of graphic notation. 2 Electroacoustic notation Several authors (i.e. [Cuypers, 1989]) have noted that musical thinking is very different from the sequential thinking used in conventional programming. Therefore, any graphical user interface for music composition should not present the composer with information in the sequential, essentially non-musical manner in which it is stored by the computer. This is particularly applicable to electroacoustic music, in which a vast variety of non-linear methodologies may be employed in the composition process. The relationship between graphical forms of control for electroacoustic music and the actual sounds are likely to have a more abstract character than that permitted by traditional notation. 3 A design for electroacoustic music composition The maximum cognitive association of sound to image will result from the mapping of an image created entirely by the user. The aural qualities of the sound are then associated with the visual icon created by the user in a strong cognitive bond. This is the core philosophy taken in the design and implementation of a program created for electroacoustic music composition [Rossiter, 1991] on the Composers Desktop Project [Atkins et al., 1987] music workstation. 4 A two-tiered approach In the implementation of this system, a two-tiered approach has been adopted. An underlying syntactic control system [CDP, 1993] provides the integration and flexibility required for low level soundfile control, while a highly intuitive and interactive graphic interface presides on top which can act to abstract the user away from the potentially confusing technical details. Compositional units are represented in a form which facilitates instant recall and memorability. The user is then free to think in musical terms and apply these thoughts directly within the system. In this system, the user has absolute control over the icons used to represent sound objects and the subsequent change of icon image as the sound unit is transformed. The style of visual content may employ standard musical representations, such as staves and notes, or use any other visual arrangement. In this way, the GUI is passed in part to the user. This principle has been extended so that the graphical area in which these icons were held became an actual record of the development of the composition - in effect, a score. To this extent, Music Graphics 272 ICMC Proceedings 1994

Page  273 ï~~Four instances of one sound object - - - - - - - - - - - - - - -- Drop-down snenuw for high-level sound/image operations Scrollable window into worktop area r J } Marker showing worktop area displayed in scrollable window relative to all worktop space Display of summary screen Figure 1: Screen display further features including the ability to add lines and to label the time axis have been developed so that the user can arrange the icon area for display in exactly the manner s/he wishes, down to the choice of size of font used to label the time axis. 5 Main system display Figure 1 shows a screen display of the main system screen. The display is separated into three main areas. General system functions are accessed through a selection of drop-down menus at the top of the screen. The worktop area in which most aural/visual interaction occurs occupies the majority of the screen. At the bottom of the screen a small display shows the entire length of the worktop in summarised form. This enables the user to see at a glance the entire score progression. A marker also indicates exactly the location at which the window onto the worktop is positioned. 6 Menu selection In order to distinguish between operations concerning the iconic representation of the sound and sound transformations, two separate menu systems have been devised. In order to access the operations concerned with image manipulation, the user presses on the left mouse button. In order to access the operations concerning sound operations, the user clicks on the right mouse button. The menu options that subsequently appear are context sensitive. For example, operations applicable only to a MIDI object appear only when Edit HIDI Purge sgsten Squeeze SlSten Play soundfile Play HIDI file Set tisescale Score options Create score Play all score Play sf score Play H ID score Figure 2: Menu options for sound control the user has clicked on a sound object which has been mapped to MIDI information. Similarly, operations applicable only to soundfiles appear only when the user has clicked on a sound object which has previously been mapped to a soundfile. A third sequence of menus consists of functions which do not directly apply to a single sound object. Examples include operations concerned with printing or saving. These operations are accessed through the sequence of drop-down menus at the top of the screen. An example is shown in figure 2. 7 Sound operations The user is able to map sound to an iconic image. This may be a MIDI sequence of information, imported via Standard MIDI files, or recorded audio sound in the form of a soundfile. The user, after indicating that he wishes to map sound onto ICMC Proceedings 1994 273 Music Graphics

Page  274 ï~~an image, must then choose between MIDI and soundfile forms of information. The appropriate file selector will then appear and the user may then select a sound filename. Distinctions between the actual treatment of the two contrasting types of sound information within the system are kept invisible to the user as much as possible. When the user wishes to work on alteration to properties of either sound or iconic image, process control is passed to an external program. Different programs are mapped for the two contrasting formats of sound information. When the user has finished working in the environment of the external program control returns to the main program and the new sounds and images may then be incorporated into the composition. The standard MIDI file format was chosen for representing MIDI information so that the majority of MIDI transformations could then be left to professional MIDI software such as 'Notator' and 'MidiGrid'[Hunt, 1990]. Sounds can additionally be manipulated within the main program itself in the following way: " Soundfiles The user has control over a number of soundfile attributes, including the gain and the stereo position (if applicable) of any sound object that is mapped to a soundfile, " MIDI The user is able to transpose any MIDI sound object up or down in pitch, as well as alter the tempo of the sound. Both MIDI and Soundfile sides of the composition are run concurrently in the final mix. However, both are playable separately. This is useful both for a composition which uses only one of the two sound formats, and for assessing either format in solitude for reasons of equipment testing. 8 Graphical facilities Features have been developed to enable the user to arrange and further improve the visual arrangement of the compositional area so that the display may be tailored as a score in exactly the required manner. This includes the following items. " Vertical lines The scores of electroacoustic music employ the use of lines to break up or mark out sections within a piece (i.e. [Johnson et al., 1967]). The evolution of scores from standard notation may have eclipsed use of the traditional 'bar', but equivalent uses of lines are still prevalent in twentieth century notation. A feature was implemented so that composers using the system could similarly employ line demarkation. For vertical lines, the user is able to select the time interval at which they appear. Subsequent alteration of this interval may be made, resulting in an automatic redraw of the bars in their new visual locations. Bars may be deleted singly or all at once. Desk File Sound lmbers Lines Print Path i i ti ---- - - ---------- - Figure 3: Worktop with line demarkation " Horizontal lines Similarly, horizontal lines may be used to aid in the the visual structure and spacing of the score. A strict vertical interval is not enforced between lines of this orientation. The user is able to control the position of the line simply by selecting the appropriate option. A horizontal line appears, and the user moves the mouse up and down and clicks the mouse button when the desired vertical level is reached. " Line pattern To give the user more control over the appearance of the final score, a dotted pattern of any mark:space structure (up to the extremes of a solid or invisible line) may be selected for lines of either orientation. " Timebase The user may request the labelling of the time axis to denote the progression of time across the score if the score has been arranged with regard to a conventional, linear progression of time. In addition, the user can introduce an offset to the time axis, so that the score can be presented as, for example, a second movement. This supports the system in the production of interconnected pieces of a work. An example of the worktop shown in figure 1 with horizontal, vertical and timebase markings added is shown in figure 3. 9 Example use Several composers have used the system for a variety of compositional tasks. An example of a composition in the intermediate stages of construction is shown in figure 4. The composition is based on recordings taken from a train transport system. In this instance the composer has used icons of a temporary nature which are sufficient for a strong aural-visual bond during the compositional process with a view to later refining these images as part of an integrated visual arrangement more representative of the composition as a whole. Another example is shown in figure 5, in which the composer has created a composition based on military noises from land, sea and air. The composer Music Graphics 274 ICMC Proceedings 1994

Page  275 ï~~rile Ines Irint 111sa:l: 4il iI l, ' I,11 I 4 t rated witini his comition intoione egoies,:i ndi by the I 'tre hoizonitalband. airsoundi hiscomosi.tIinwa created. en iyfo mi i filesi frm of- bief ma i theione et to incatpe sthe transltion in threedimsioal spaced thsoe oundsl sunderwentrpat rted fithi exampilstraites an otrele diseorred, ndiualldy utrtred aorronVan n. TConsnglyin the sertcndr exampe te composer has employed6 1ariidisaltrutue,-it ceryFined4iconsmpe see1inasetlordere way.isale cdionast betense the ocompls den trthine toh flxiiicty of tenesyste in sutegiensindiatpeththmeiontalderwntdst Thn s e wristhanypriulrbneredsge porti anumer oftdiferntes arcetswdselderoaoundT cis c sition. waratementirely fmedmbionic pres.ntedform noflybrieratemaay. Teprsonasbeteensd the oexmpse inteon st thefiindictythe ssltonin tsredimeninalnumethomfeesntdpproarhestatd eCentrastilycompoestiondxmltecm ie i i mu s e i:iPrt 1Paths - ip. S.. 1 ii...T.... i i.i.. ' I,, | 11.:111.:.11 i i~ i iV: ' ' I i i l f il l isi Figure 5: Example score 2 (sections 1 and 2) 12 Acknowledgements This project benefitted from the good advice of many people including Sean Watkinson, Andy Hunt, Richard Orton, and Tim Anderson. A considerable debt is owed to them. Thanks must also go to Rupert Till for permission to use his compositions for this paper. References [Atkins et al., 1987] M. Atkins, A. Bentley, T. Endich, R. Fischman, D. Maham, R. Orton, and T. Wishart. The composer's desktop project. Proceedings of the International Computer Music Conference. Urbana, Illinois, 1987 [CDP, 1993] Groucho manual, Composer's Desktop Project documentation, York, 1993 [Cuypers, 1989] Cuypers, L. Computer Music Programming. 14th Euromicro Symposium on Microprocessors and Programming, 25:1-5, pp65-91, Jan 1989 [Hunt, 1990] Hunt, A. MIDIGRID: An innovative computer-based performance and composition syistemn. Proceedings of the International Computer Music Conference, pp392-394, 1990 [Johnson, 1967] R.Johnson. Scores: an anthology of new music. Schirmer books, 1967 [Rossiter, 19911 Rossiter, D. Score: a graphical environment for electroacoustic composition. Unpublished Masters degree report, York University, England, 1991 10 Conclusions An environment has been created for electroacoustic music composition in which the user is able to control both the aural product and the corresponding visual representation. The system is designed so that the graphical arrangement may be extended in the production of a notational score. Support is provided for low-level control through the provision of access to an underlying syntactic control system. Trial use by composers have highlighted the success of the concept. 11 Further information The original MSc thesis upon which this paper is based [Rossiter, 1991] is available through anonymous ftp transfer from, in directory /pub/distrib/musictechnology/masters_reports, as file ICMC Proceedings 1994 275 Music Graphics