~INTEGRATED MULTIMODAL SCORE-FOLLOWING
ENVIRONMENT
Martin Ritter
School of Music
University of British
Columbia
Vancouver, Canada
Keith Hamel
School of Music
University of British
Columbia
Vancouver, Canada
Bob Pritchard
School of Music
University of British
Columbia
Vancouver, Canada
ABSTRACT
The Integrated Multimodal Score-following
Environment (IMuSE) [5] is a software project aimed at
developing a system for the creation, rehearsal and
performance of score-based interactive computer music
compositions. An enhanced version of the
NoteAbilityPro [7] music notation software is the central
controller in IMuSE. The score contains conventional
notes and performance indications as well as discrete and
continuous control messages that can be sent to other
applications such as MaxMSP [15] or Pure data (Pd)
[11] during performance. As well, multiple modes of
score-following [2] can be used to synchronize the live
performance to the score. Score-following strategies
include pitch-tracking of monophonic instruments, pitchtracking and amplitude-tracking of polyphonic
instruments and gesture-tracking of performers' hand
movements.
In this paper, we present an overview of the IMuSE
system, with a focus on its abilities to monitor and
coordinate multiple pitch and gesture trackers. As an
example of gesture-tracking, we show how video
analysis is used to follow piano hand movements. We
discuss strategies for negotiating between multiple
streams of score-following data and we assess the
strengths, limitations and future research directions of
the IMuSE system.
1. BACKGROUND
Score-following has a relatively long history, with the
first research being done in the early 1980s by
Dannenberg [5, 6] and Vercoe [24, 25]. This research
was aimed at automated accompaniment where the
computer would generate midi or audio tracks that were
synchronized to a live musical performance. In the
following decade, Puckette [17, 18] developed the
explode Max object that could be used for monophonic
score-following of pitch-tracked or midi data. In the
early 2000s, score-following systems such as suivi- [20,
21] and antescofo- [4] were developed using Hidden
Markov Models [1]; these systems prove to be far more
robust in performance situations where machine
perception and performer mistakes need to be
accommodated. More recently, audio analysis has been
used for music retrieval systems such as Query-byHumming [12] and intelligent audio editors [7]. Our
focus is on score-following in live interactive computer
music compositions involving several performers and
where complex audio interactions between the live
instruments
required.
and the computer generated sounds are
2. INTRODUCTION
One of the challenges of creating score-based interactive
computer music compositions is that there is often no
connection between the notated score and the
programming environment used to produce the
electroacoustic layers of the composition or to process
the live instruments. As a result, composers must
develop strategies for synchronizing the live and
electroacoustic components of their composition using
manual cues, foot pedals or rigid timing controls. As
well, the fact that several disconnected software
applications are being used in an interactive performance
can make rehearsing difficult and the editing of the
composition can be awkward since it often means
altering several different software components.
The Integrated Multimodal Score-following
Environment (IMuSE) has been developed to alleviate
some of the issues associated with creating, rehearsing
and performing score-based interactive computer music
compositions. While this system could be used in some
improvisatory performances, it is primarily designed for
compositions that have a well-defined score and a where
a high degree of synchronization between the live
performance and the electroacoustic processing is
needed.
3. SYSTEM OVERVIEW
The central software component in IMuSE is the
notation program NoteAbilityPro. This application has
been enhanced to allow MaxMSP messages,
multichannel breakpoint functions (BPFs), extended
notes, conTimbre [9] controls and antescofo- action
messages to be added to the score and aligned to other
score events. As well, NoteAbilityPro can be externally
controlled through network UDP messages so that
synchronization information from pitch-tracking and
gesture-tracking of a live performance can be used to
alter the playback of the score. In order to accommodate
conventional playback methods, remote messaging and
synchronization with the live performance, music staves
in a NoteAbilityPro score can have different functions:
they can contain music notation that is intended for
performers to play, notes and sound files that are
intended to be played as audio events directly from the
score, messages that are intended to control external
applications, or notation and continuous graphical data
185 2013 ICMC
