Page  00000088 KEYed user interface: Tools for expressive music production Farhan Mohamed, Sidney Fels Human Communications Technology Laboratory, Department of Electrical and Computer Engineering, 2356, Main Mall, University of British Columbia, Vancouver, BC, V6T 1Z4 fmohamad@hotmail.com, ssfels@ece.ubc.ca Abstract We introduce the KEYed user interface, an ergonomic and intuitive method for controlling music composition software from a piano controller by adding a momentary foot switch. The piano keyboard mappings provide increased speed and intimacy with the controls for enhancing the expressivity of the composer during the composition. After describing the current practices and design of the system, we discuss the results of our early user testing during various editing tasks, comparing the conventional input device with ours. 1 Introduction Modem music composers use different computer tools to aid their creative process. Many tools are used for recording the processes of music making, including sonic materials. The use of these tools is mostly due to the modem trend of composing with complex production functions in mind. Further, these production functions have become an integral part of the ideas and constructs of the composer's mind, which are externalized during composing and are critical in the overall output. For example, spatial positioning of a sound is not something which is added post-synthesis, but plays an equal role along with other parameters such as fundamental frequency, formant frequency, and so forth (Clarke 1999) during composition. New interfaces for musical expression are typically thought about in the context of real-time performances. However, we contend that the functions and the inherent complexity in the modem composition process provide opportunities for new ways of expression. Functions such as the spatial positioning, fades and mutes of a sound source are performed real-time during composition. 1.1 Composers And Recording Consoles Industry standard sequencing packages, such as Cubase and Cakewalk, allow the composer to perform those production tasks commonly done by an audio engineer, but in a cheaper setup. Typically, composers record a section of a track and then overlay the musical parameters associated with the track on a separate MIDI channel, or automate them directly on to the software mixing-console's faders and knobs. The modem composer is thereby required to learn the functions and signal flow of a complex mixing console, as shown in Figure 1. Figure 1: Solid State Logic Avant mixing console These functions, which require high precision and expressivity, are performed by the composer while playing back the pre-composed track in real time. Further, other editing functions such as performing cuts, punch in/out, and setting locators also contribute to the overall expresiveness of the composer. Coupling these facts together, we realize that new interfaces to perform such production functions may greatly enhance the musical expressive abilities of the composer. 2 The Need For A New Interface Due to the complexities of sound processing during music composition, existing user interfaces for sequencing electronic music have become cumbersome, as they require the use of multiple input devices like a piano controller, a computer keyboard, and a mouse. Synthesizer software such as CSOUND, features the ability to map a limited number of MIDI events to keys on a piano controller. Standard electronic piano controllers have additional sliders, knobs, and dials that can be programmed to trigger MIDI events. Systems such as the E-scape, which was designed for the disabled, shows some interesting variations on these by using one or more switches, and flexible use of the existing mouse or console keyboard, by providing 88

Page  00000089 options using menu selection (Anderson and Smith 1996). One disadvantage of these conventional methods is the excessive physical space they occupy. More importantly, the musician's creative work is constantly interrupted by the time spent switching between multiple input devices and figuring out their functions. KEYed user interface provides a mapping of these control functions from the piano keyboard. This is done using an octave structure, with key based segmentation. We intend to add a touchpad for fine control of selected continuous parameters. This customizable mapping provides the musician with a familiar configuration of space and sound, allowing him or her to focus on the creative aspects of music composition. While the touchpad is a new device for the composer, we are carefully placing it to minimize the acquisition time as discussed below. 3 KEYed ui System For many computer composition functions, KEYed user interface eliminates the need for the computer keyboard and mouse by relocating their functions to the piano controller. An example of a macro that can be relocated to the controller is the copy function, or [Control]-[C], which copies a selected sequence to the clipboard. To distinguish between keystrokes that represent notes and those that represent macros, a momentary foot switch is used as a mode switch, thereby reducing mode errors (Sellen, Kurtenbach, and Buxton 1992). Though a secondary body channel, such as the foot, has a lower information processing bandwidth, it sets the framework and the reference for the primary body channel and the task in this system, as shown in Figure 1. setup among composers. Macros from the different Cubase windows were mapped to specific octaves on the controller keyboard for separation. 4.1 Keyboard Layout Design The design of the layout incorporates the four windows most commonly used by the composer. The functions associated with each window map to a specific octave on the piano controller keyboard. The piano keyboard has been laid out as follows: * The octaves used are the central ones found on every standard music keyboard. This layout is shown in Table 1. WINDOW OCTAVE USED Transport bar C3 to B3 Arrange Window C4 to B4 Key Edit Window C5 to B5 File Handling C6 to B6 Table 1: Window-Octave Layout * The middle four keys (E, F, F# and G) are used either for going up or down the tracks or parts, or punching in or out, as shown in Figure 3. They correspond to the computer commands '-', '1', '1T', '--'. The first note, 'C', of the different octaves is designed to 'open' windows. * The remaining mappings are designed in accordance with the timbre associated with the notes. * The highest note, 'C7' is used as an 'Enter' key for all confirmation functions. We anticipate that musicians can conveniently hit this key as it is at the extreme end and hence, doesn't need to be looked for. For example, the transport octave is used to play, stop, record, mute, and solo a sequence or a specific MIDI channel. Record Stop Punch-In Left locator & A& 4 Right locator C3 B3 Music Notes, Computer Commands S--- * Mode Switch Secondary Channel Primary Channel (Foot) (Hand) Figure 2: KEYed user interface system setup In addition, the system state is reinforced by the foot proprioceptive feedback, and the relocation of the functions to a single input device minimizes the device acquisition time (Sellen, Kurtenbach and Buxton 1990). 4 Design We chose to use the Cubase sequencing program by Steinberg on Microsoft Windows, which is a popular Open/ Close Start Rew Fwd Solo Click Punch-Out Figure 3: Transport Octave * For continuous parameters, we intend to add a single point sensitive touchpad. The touchpad is used for constrained vertical or horizontal actions, for a single degree of freedom (DOF) task like sliding a fader, and to perform a full 2DOF task like drawing a modulation graph. A double note combination selects parameters like the volume, pan, effects, modulation, pitch shift and more. Finer controls of these parameters are 89

Page  00000090 performed using the single point sensitive touch pad, which is integrated on to the top of the piano controller for faster acquisition, as shown in the Figure 4. For example, depressing 'C3' and 'E3' simultaneously selects the volume for the selected track, and a volume graph can then be drawn using the touch pad. The key combinations were designed with the composer's familiar fingering patterns in mind. 4) g 140 120 100 80 60 40 20 0 0- Computer keyboard Sm Rano controller Test 1 Test 2 Akl Figure 4: Real Time Editing with KEYed 5 Experiment Design For our preliminary user studies, two interaction methods were tested: the Computer Keyboard and the Piano Controller. Tasks were designed for functions which are commonly perfomed using the computer keyboard. Hence the touchpad was not needed in our early experiments. A total of 6 subjects with sequencing experience performed the tests with these two methods in a preassigned order. Subjects were given a practice run to explore different modes and strategies. To test for the mean completion time, two tasks involving repetitive sequencing and editing were performed twice, thereby requiring the subjects to switch back and forth between the modes on their own. Some of the components of the tasks used in our experiments included opening windows, setting locators, recording a section of track, soloing a specific track, creating new tracks, performing cuts and pastes of the recorded track, and so forth. We chose to turn the sound on while the piano controller was in the control mode in order to observe the effectiveness of the associated earcons (Mynatt 1994). After the tests, the subjects were asked to rate the input devices on a 0 (terrible) to 6 (great) scale, based on their experiences. During testing, we explicitly checked for any mode errors while using the piano controller. 6 Evaluation With an average learning time of approximately 5 minutes, subjects performed both tasks significantly better with the piano controller in Testl (p<0.05, paired t-Test). Although Test2 (p<0.06, paired t-Test) was done faster than Test, as shown in Figure 5, the performance was not significant with the piano controller. We suspect Figure 5: Mean Time with 95% Confidence Error Bars this was due to the minimal learning time, and small number of subjects used for the study. Subjects gave the computer keyboard a significantly lower rating than the piano controller (p<0.021, t-Test). They found the piano interaction intuitive and an effective link between playing and editing. Though the no-sound condition was not tested in our study, from the questionnaires we learnt that the earcons (Mynatt 1994) were effective for performing the tasks. This was because subjects had already acquired auditory familiarity with the piano. Such a feature could facilitate complete control and use of the system by composers with visual disabilities. Interestingly, no mode errors were detected during testing using the piano controller. We suspect that this is due to the subject's familiarity with damper and sustain pedals, which are momentary pedals as well, and are commonly used in the industry. 7 Conclusion In essence, the functions and the inherent complexity in the moder composition process provide new ways of expression. Expressive controls for these functions can benefit from new interfaces such as the KEYed user interface. The KEYed user interface also illustrates how an appropriate mapping of layout, feedback, and context is important in the design of modem composition user interfaces. The system leverages on the musician's spatial and auditory memory of the piano keyboard. Features for complex sound editing and control are integrated into the system; therefore the user interface requires far fewer operations to achieve various production tasks. This helps the composers focus on musical rather than operational issues. From our early user studies, we found that the current computer keyboard interface impairs the flow of creative ideas for expression. 8 Future Directions This system is undergoing continuing developments, which will allow it to expand in its range of usefulness. Our future research includes * Integrating a touchpad to the existing system * Continuing user studies 90

Page  00000091 * Applying the concepts of KEYed user interface to other musical instruments * Exploring musical structures for good mappings * Investigating different key boundaries 9 Acknowledgements We wish to thank Lorna Slater and Edgar Flores for their input to the system design, ATR MI&C for the fiscal support, and the volunteer subjects for their valuable time. References Sellen, A., Kurtenbach, G. and Buxton, W. 1992. "The Prevention of Mode Errors Through Sensory Feedback." Human-Computer Interaction, v.7 n.2, pp. 141-164. Sellen, A., Kurtenbach, G. and Buxton, W. 1990. "The Role of Visual and Kinesthetic Feedback in the Prevention of Mode Errors Interactive Technologies and Techniques." Proceedings ofHuman-Computer Interaction, pp. 667 -673. Zhai, S., Smith, B., and Selker, T. 1997. "Dual Stream Input for Pointing and Scrolling." Proceedings ofACM Conference on Human Factors in Computing Systems, v.2, pp. 305-306. Anderson, T., Smith, C. 1996. "'Composability': widening participation in music making for people with disabilities via music software and controller solutions." Proceedings of the ACM conference on Assistive technologies, April 11 - 12, Vancouver, BC. Clarke, M. 1999. "Composing with multi-channel spatialization as an aspect of synthesis." Proceedings of International Computer Music Conference, pp. 17-19. Mynatt, E. 1994. "Auditory Presentation of Graphical User Interfaces." Auditory Display, Ed. Gregory Kramer, SFI Studies in the Sciences of Complexity, Proc. Vol. XVIII: Addison-Wesley. 91