EVALUATING THE WIIMOTE AS A MUSICAL CONTROLLERSkip other details (including permanent urls, DOI, citation information)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. Please contact firstname.lastname@example.org to use this work in a way not covered by the license. :
For more information, read Michigan Publishing's access and usage policy.
Page 1 ï~~EVALUATING THE WIIMOTE AS A MUSICAL CONTROLLER Chris Kiefer Dept. of Informatics University of Sussex Brighton, UK Nick Collins Dept. of Informatics University of Sussex Brighton, UK Geraldine Fitzpatrick Interact Lab Dept. of Informatics University of Sussex Brighton, UK ABSTRACT The Nintendo Wiimote is growing in popularity with musicians as a controller. This mode of use is an adaptation from its intended use as a game controller, and requires evaluation of its functions in a musical context in order to understand its possibilities and limits. Drawing on Human Computer Interaction methodology, we assessed the core musical applications of the Wiimote and designed a usability experiment to test them. 17 participants took part, performing musical tasks in four contexts: triggering; precise and expressive continuous control; and gesture recognition. Interviews and empirical evidence were utilised to probe the device's limitations and its creative strengths. This study should help potential users to plan the Wiimote's employment in their projects, and should be useful as a case study in HCI evaluation of musical controllers. 1. INTRODUCTION Nintendo's Wii gaming console is enjoying considerable success; in 2007 20 million were sold worldwide, more than any of its rivals1. The reason for this popularity could be partially attributed to the innovative design of its controllers, whose motion sensing capabilities introduce gestural control into gameplay. The console's principal controller, nicknamed the Wiimote, can function independently from the Wii. It sends data wirelessly using the Bluetooth protocol, which means that ordinary computers can read its output. For our interests here, this output can in turn be directed to audio software, enabling the device to be employed as a musical controller. Musicians' growing interest in the Wiimote can be seen from the proliferation of demo videos and community sites on the internet 2. People have found creative applications for the controller which include drumming, DJing, synthesiser control and more. This interest is reflected in the audio software community, with Wiimote extensions available for many audio environments. Generic solutions such as GlovePIE 3 or DarwiinRemote OSC4 allow the conversion of Wiimote data to MIDI or OSC. 1 Console wars. http://en.wikipedia.org/wiki/ Consolewars 2 e.g. WiiLi. http://www.wiili.org/ 3 http://carl.kenner.googlepages.com/glovepie 4 http://code.google.com/p/darwiinosc/ Given that the device is in use by musicians and yet wasn't specifically designed for this purpose, there is an opportunity now to investigate the Wiimote in an attempt to gain a better understanding of its capabilities in a musical context and how its use might be evolved with music in mind. To this end, we carried out an evaluation experiment of the use of the Wiimote as a music controller. We looked to HCI to provide methodology for conducting the experiment. As well as wider HCI literature [3, 5], we drew on Wanderley and Orio's  research into evaluating musical controllers, and Hook's  study on evaluating usability in interactive art. 2. THE WIIMOTE To determine how to evaluate the Wiimote, we need to examine the musical possibilities offered by its output data. The Wiimote embodies three types of control: a threeaxis accelerometer for motion sensing, an infrared camera for pointing, and various buttons. For the purposes of our experiment we concentrated on the accelerometer, a rare feature in musical controllers and arguably the most interesting feature for musicians. The IR camera works in conjunction with a sensor bar that connects to the Wii via a proprietary connector, and it requires some electronics expertise to customise this to connect to a home computer5. Probably as a result of this it doesn't seem to be widely used for music, so we excluded it from the evaluation. The Wiimote's physical feedback 'rumble' function was also excluded from the study as its timing resolution is arguably too low for musical purposes. The effect of gravity on an accelerometer  means that it can be used to measure rotation about the Z (roll) and X (tilt) axes, though yaw cannot be measured as gravity has no bearing on the device when rotated about the Y axis. These roll and tilt readings can be used for continuous control, with two caveats; firstly, accuracy only comes from pure rotation, additional motion causes acceleration that will add noise to the data. Secondly, the output is not precisely linear, approximating instead to a slight 's' shape with values bunching up in the centre. The raw acceleration data can be interpreted in different ways. One possibility is for triggering; a peak de5 USB Wii Sensor Bar. http://www.modpulse.com/articles/consolemods/usb-wii-sensor-bar
Page 2 ï~~tection algorithm can determine when a drumming-like motion has been made, and in turn set off an event such as playing a drum sample. Another possibility is gesture recognition, which presents several challenges. Acceleration data from the Wiimote is inherently noisy; if it is rolled or tilted then gravity affects the readings and it is difficult to remove this component to get the true acceleration value. Conversely, a rotation may look like an acceleration movement, but the Wiimote hasn't actually moved. For this reason, it is difficult to determine absolute position reliably, so gestures must be inferred from the raw acceleration values. Neural networks  are well suited to solving this sort of classification task. Having examined the basic capabilities of the Wiimote for musical control, we planned an experiment to test them. 3. THE EXPERIMENT 3.1. Metaphors and Musical Tasks Following guidelines from , the experiment comprised asking participants to perform simplistic musical tasks within metaphors which were chosen to test the basic capabilities of the Wiimote. We wanted to run a comparative study, asking participants to perform these musical tasks on an additional controller other than the Wiimote. We chose the Roland HPD-15 HandSonic 6 for this purpose as it provided the interface elements that might typically be used for performing these equivalent tasks. We shall now describe the metaphors and musical tasks, and the data collected for each: 3.1.1. Triggering Participants were asked to trigger drum samples by making drumming-like motions with the Wiimote. They played simple patterns in time with a metronome beginning with crotchets, moving up to quavers and then any pattern they chose. They performed the same tasks using the HandSonic's drum pads. Drum trigger events and metronome beats were logged for later analysis. 3.1.2. Precise Continuous Control Continuous input controlled the pitch of a saw wave in 6 discrete steps. Participants were asked to move up and down though successive pitches in time to a metronome, using both the roll and tilt axes of the Wiimote and also a knob on the HandSonic. The pitch changes and metronome beats were logged. 3.1.3. Expressive Continuous Control This task involved simultaneous control of the grain density and filter parameters of a generated sound. These parameters were mapped to the roll and tilt axes of the Wiimote, and to two knobs on the HandSonic. Participants experimented with each controller for about two minutes. 6 http://www.roland.co.uk/drumroom_catdet.asp?ID=HPD15 The less defined nature of this and the next context would not lend themselves well to statistical analysis; no data was collected for either. 3.1.4. Gesture Recognition A multilayer neural network was trained using back propagation to recognise 5 different shapes, which could be drawn while holding down the B button on the Wiimote. Five rhythmic musical tracks were running simultaneously; each shape was assigned to a track and recognition of the shape muted or un-muted this track, quantised to the nearest bar. Participants were given a printout of the gestures and asked to play with the system for 3-4 minutes. There was no comparison controller in this part of the experiment. 3.2. Method and Implementation Each session of the study started with the participant being asked about their musical and Wiimote experience. Before each context they were given a period of practice with each controller, then afterwards they were interviewed about their experience. The order of the controllers and Wiimote axes was swapped between participants to reduce learning effect. After the first three contexts we questioned them about which controller they preferred and why, and asked them to describe the advantages and disadvantages they felt the Wiimote possessed in that particular metaphor. With no comparative controller, the interview after the gesture recognition context was more open ended; participants were asked to comment on their experience and were questioned on certain aspects of the task. Having completed the tasks, they were asked some general questions about their experience. The software for the experiment was programmed using the SuperCollider audio programming environment, which was connected to the Wiimote via DarwiinRemote OSC and to the HandSonic via MIDI. This software also recorded data logs. Participants were videoed in order to observe how they used the Wiimote and to record their answers to the interview questions. The logged data was analysed in MATLAB (as later described), and a qualitative analysis of the video interview data was conducted, identifying and coding common themes. There were 17 participants, with an average age of 31.2 (min: 20; max: 46). For the purpose of statistical analysis, nine were classified as musicians based on a combination of years of study and practice routine. They had either completed at least six years of training, or at least two years and were practising at least six hours per week. The musicians averaged 7.7 years of study (min: 2; max: 15) and 6.8 hours of practice per week (min: 0; max: 25). Participants also rated their Wiimote experience on a scale of zero (none) to five (high); most participants had no prior Wiimote experience; we classified garners as having two or more for this value, giving us six garners with an average experience of 3.2.
Page 3 ï~~I I I Triggering Precise Cont. Ctrl Expressive Cont. Ctrl L Wiimote 18% 35% 53% I HandSonic 70% 53% 23.5% Neither 12% 12% 23.5% Table 1. Controller Preferences 4. RESULTS 4.1. Statistics Experimental data was analysed using ANOVA and posthoc t-tests with respect to factors of Wiimote/HandSonic, musician/non-musician and gamer/non-gamer as appropriate. Table 1 shows the participants preferred controllers for the first three tasks. There was no significant overall preference for either controller. For the triggering experiment we logged the timing of each successful trigger in the crotchet and quaver tasks, and analysed the data to determine the average timing error relative to the task requirement. ANOVA tests revealed no significant difference between the Wiimote and the HandSonic, both overall and between sub-groups of musicians/non-musician and gamners/non-gamer. Pitch changes during the precise continuous control experiment were analysed to obtain the frequency of changes and timing errors for each of the roll axis, the tilt axis and the knob on the HandSonic. There was a borderline nonsignificance between the number of pitch changes from the roll and tilt axes (p=0.0542), pointing towards a tendency for the tilt axis to be more accurate for control. Comparing the roll and tilt axes to the knob via individual t-tests gave highly significant differences (p<0.00001, p=0.0003 respectively), which was expected considering the stability of a knob compared to the Wiimote. We found no significant difference in timing error between the three control methods. 4.2. Interview Results 4.2.1. Triggering The main focus was on the lack of physical feedback with the Wiimote. The absence of a real contact point made the task difficult for some; one participant solved this by drumming against their hand. Some people commented on the intuitive nature of the Wiimote during this task ('it has the feeling of a virtual drum stick'), and many talked about the benefits of the controllers portability. It was observed that some participants found difficulty in drumming faster rhythms, especially semiquaver patterns. 4.2.2. Precise Continuous Control Issues emerged mainly in the areas of control, mapping, ergonomics and feedback. People commented that they found it easy to get stuck between notes, that was it difficult to judge the boundaries, and that the Wiimote was generally less steady and precise than the HandSonic's knob. On the positive side, some liked the speed and freedom of movement compared to a knob. Several participants noticed how the control wasn't precisely linear. Ergonomically, some people found the 3600 rotation action unnatural for their wrist. Participants talked about the lack of visual feedback with the Wiimote, preferring the HandSonic where they had a viewable reference for the controller setting. 4.2.3. Expressive Continuous Control Fun was a prominent theme here, some people feeling that the Wiimote was a more enjoyable way of controlling the sound, especially as precision wasn't required. Many participants talked about the intuitive or embodied feel of using the Wiimote in this context with comments such as 'it's almost like your own hand making the noise', 'I had to think a lot more using the HandSonic' and 'it's more instrument like, less computer like'. Expressiveness and musicality was a common topic; some felt it gave more possibilities ('you can explore stranger noise combinations') and more room for expression. People observed that you could control the parameters percussively, and some enjoyed the randomness the Wiimote added to the control process. In terms of control, there were comments about repeatability ('it's hard to stay in one place, which could be good or bad depending on context') and co-dependence ('it's easy to affect one parameter while changing the other'). Several people said they appreciated controlling two parameters with one hand. Physical feedback was mentioned again - 'I prefer the hard limits of the HandSonic'. 4.2.4. Gesture Recognition Participants again emphasised the fun aspect of using the controller ('it's fun way of turning stuff on and off', 'keep fit with drum loops!'). Opinions were mixed on the subjects of intuitiveness ('it felt like a biological relationship between me and the music', 'it was strange to control with shapes, I didn't really feel part of it') and expressiveness ('didn't feel more expressive than playing a button', 'making a shape feels more dramatic and connected to the music'). One participant observed that this task was the most analogous to Wii gaming. In terms of gestures, several people would have preferred to make smaller more subtle gestures, and some said they would have liked some sort of continuous measurement of the gestures, rather than just binary control. One participant would have liked less arbitrary gestures that related more to the sound. People thought that this kind of application of the Wiimote would be good for performance, and would look good on stage. 4.2.5. General Comments Participants were asked about what expectations, if any, they had of the Wiimote in terms of music. The majority of people said they had none, and two said they thought it would have been less responsive. When asked whether
Page 4 ï~~they imagined any sort of metaphors while using the Wiimote, most participants said they didn't. Some people imagined the Wiimote as a drum stick in the first task, although this was problematic for one person - ' it makes you want to use it like a drum but it's not like a drum. if you think of it as a drum stick, it's difficult to get it to do what you want'. One person thought of rolling the controller as being like turning a knob. About whether they could imagine using the Wiimote in their own projects, several people thought it would make a better tool for performance than composition - 'I'd use it to make performance more of a spectacle'. Guitarists talked of attaching it to their guitars for an extra dimension of control. A singer proposed using it for controlling vocal effects during performance. Other people suggested strapping it to limbs or using it for conducting. Participants were also asked for how they thought the Wiimote could be improved, either physically or functionally. A strong theme was absolute positioning, participants believing that this would make the device more useful. Another theme was virtuality, participants wanting the device to have some sort of physical feedback or visual feedback such as a laser pointer at the end. The shape of the device was an issue 'it's like a TV remote control', with suggestions for a malleable surface or a rubber grip. Someone commented on the weighting - 'weighting and balance is important, different sounds need different weightings'. In terms of additional controls, there were suggestions that larger buttons would be better for music and that some sliders would be useful. 5. DISCUSSION We set up this study to explore more systematically how the Wiimote functioned as a music controller, comparing it to another type of controller both in terms of actual performance and user experience. While the statistical analysis showed no significant differences, there were some interesting themes arising from the interview data. Two overall themes emerged: virtuality and expression. The abstract nature of interaction with the Wiimote seemed to be the main contributor to control issues that some participants had with the device: lack of hard limits for continuous control, lack of physical feedback, lack of visual reference and lack of a concrete metaphor. One participant commented on how this abstraction made the triggering task difficult - 'the virtual nature of the Wiimote makes it harder to keep the rhythm in my head, after a while I started to lose the meaning of what I was doing'. This virtuality is also one of the Wiimote's strengths, giving it flexibility for use in multiple contexts and providing one of the roots of the embodied and intuitive experience some participants observed during the study. The expressive continuous control context was the only metaphor where the Wiimote won the title of 'most pre ferred controller', and perhaps this points to where some of the Wiimote's strengths lie for musicians. The physical nature of accelerometer control lends a natural random ness to its output, and the addition of acceleration motions such as 'flicking' the controller add an extra and slightly unpredictable dimension to the possibilities. This manner of less precise control sits very comfortably in the context of creative or expressive sound manipulation. This is not to say that the Wiimote isn't useful in more structured contexts. From a functional point of view, there was no overall difference in timing error between the two controllers for the time critical tasks, so the Wiimote was just as capable as the HandSonic for timed accuracy at the resolutions we measured (quarter and eighth notes). 6. CONCLUSION We've examined the potential musical applications of the Wiimote, and evaluated its usability in what we believe to be some of the core contexts of its use. This evaluation has shed light on some of the problems that might occur when employing the device, and on situations, such as expressive control, where the controller may yield more creative potential. The Wiimote is relatively cheap and easily connectable to home computers, making it widely accessible to musicians. The results show it can add interesting and novel dimensions to musical control, provided that some limitations are accounted for. This study doesn't claim to be exhaustive, although we hope that we've covered the core functions of the Wiimote. In terms of applications for the accelerometer data, more evaluation could be carried out in the area of FFT analysis contexts such as in . Work could also be done on the combination of buttons and the accelerometer, and of the Wiimote with the Nunchuck or the sensor bar. 7. ACKNOWLEDGEMENTS The authors would like to thank the volunteers who spared time to participate in this study. 8. REFERENCES  C. Bishop. Neural networks forpattern recognition. Oxford: Clarendon Press, 1995.  B. Bruegge, C. Teschner, P. Lachenmaier, E. Fenzl, D. Schmidt, and S. Bierbaum. Pinocchio: conducting a virtual symphony orchestra. In ACE '07, pages 294-295, New York, NY, USA, 2007. ACM Press.  A. Dix, J. Finlay, G. D. Abowd, and R. Beale. HumanComputer Interaction. Prentice Hall, 3rd edition, 2004.  K. HO5k, P. Sengers, and G. Andersson. Sense and sensibility: evaluation and interactive art. In CHI '03, pages 241-248, New York, NY, USA, 2003. ACM.  J. Jacko and A. Sears. The Human-Computer Interaction Handbook. LEA, 2003.  D. Mizell. Using gravity to estimate accelerometer orientation. In Proceedings of the Seventh IEEE International Symposium on Wearable Computers, 2003.  M. M. Wanderley and N. Orio. Evaluation of input devices for musical expression: Borrowing tools from hci. Comput. Music J., 26(3):62-76, 2002.