# Inverse Performance Theory

Skip other details (including permanent urls, DOI, citation information)This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. Please contact mpub-help@umich.edu to use this work in a way not covered by the license. :

For more information, read Michigan Publishing's access and usage policy.

Page 533 ï~~INVERSE PERFORMANCE THEORY Guerino Mazzolal MultiMedia Lab, Institut fur Informatik der Universitat Zurich, Winterthurerstr. 190, CH-8057 Zurich Tel ++411 821 9856 Fax ++411 82198 51 e-mail: gbm@presto.pr.net. ch ABSTRACT. The performance transformation P which maps a score, its analysis and performance grammar to a fixed performance is far from one-to-one: The P-fibers define high-dimensional mathematical varieties. This work is concerned with the structure of the P-fibers. This is the subject of inverse performance theory: To systematically reconstruct scores, analyses and performance grammars from a given performance. Our results make use of the mathematical category of global compositions and locally differentiable transformations. The P-fibers depend on the complexity of the analytical structure, the combinatorial local-global configuration and the transformation class. 1 Introduction Inverse performance theory deals with the factors inducing musical performances. Here is a list of fundamental problems which shows that the state of the art is rather elementary: * What is a general description of performance transformations? * How are psychophysical and structural instances linked resp. distinguished? * What is the general concept of admitted analyses? * What is a performance grammar (Sundberg, J.)? * What type of score are we allowed to relate to? * On what level of reality do we observe a performance? We will offer a feasible approach and suggest concrete problems in the context defined by the performance workstation RUBATOÂ~ (Mazzola, G. & Zahorka, O. 1994b). Here, we are given a score from the classical type of the first Viennese period. Such a text describes a musical composition on a symbolic level, to be realized on the physical level. Performance then deals with the transformation process of this score into its performances by use of musical analysis of metrical, harmonic, melodic, contrapuntal etc. aspects of the given material. However, we are not going to trace a performance back into its psychological conditions. This delicate business should refer to an a priori insight into the structural variety laying above a given performance; in fact, Gabrielsson's claim (Gabrielsson, A.) that "in summary, we may consider emotion, motion and music as being isomorphic", is as universal as difficult to specify. Within RUBATO, the symbolic description level is based upon the PrediBase database system (Zahorka, 0.) which allows to do analyses via a collection of analytical RUBETTESÂ~. Suppose that a database A of analytical results has been calculated. This data may be applied to parts of the score by iterated use of shaping operators, thus producing the stemma. This is the genealogical hierarchy of local performance scores (LPS) which essentially are performance fields, together with initial performance data as well as the frames containing the kernels of sound events (Mazzola, G. & Zahorka, O. 1994b). It is the integration of the performance fields that defines the intended performance. 1. This work was supported by Swiss National Science Foundation Grant Nr.2000-039422.93/l. I C M C PROC E E D I N G S 1q95533 533

Page 534 ï~~The inverse performance problem is due to the fact that stemmatic inheritance and ramification is far from reversible: You cannot, in general, trace back the actors from their effects, even while restricting to a fixed score. This is a wild mathematical problem: Classification of all transformations relating any score to any performance (mostly an empty set, of course), means virtually classifying any mathematical category. The point of "direct" versus inverse performance is that it is not sufficient to produce an orderless bunch of performance transformations, but one is asked to describe entire systems of transformations, i.e. detailed and reliable correspondences of score, analyses and performances which may be the "rationale" for given performances. Understanding of expression means searching for its system. 2 Sexual Propagation on RUBATO The general RUBATO path from a score to its performances starts with a retrieval of relevant score predicates by PrediBase. These predicates are point sets within geometrical spaces and may be further combined by logical and geometric operators in the LoGeoRUBETTE. The recombined predicates are now fed into different analytical RUBETTES, and we get the first ingredient to construct performances: analytical weights. This geometric paradigm of musical analysis allows to instantiate mathematical operators for shaping performance. We come back to these operators in section 5. The stemma of the performance score is successively derived by the action of such a shaping operator S2 on a given local performance score LPS. This "mother" LPS is coupled by the "father" operator S2 and splits into a family of daughters S2 *a LPS = {LPS1,LPS2...,.LPSn}The daughters inherit all of the mother's variables plus their deformation or modification by the father's influence. On RUBATO, this inheritance tree may be altered ex post at every vertex of the genealogical tree: Here, inheritance never becomes a fixed historical input. The stemma is a flexible model of the multi-layered process of exercising and rehearsal. Thus, each of the stemma's operators fD is "nourished" by the given analysis A as a function Q(A). The primary mother LPS turns out to be the primavista performance as it is defined by the score's predicates. This being, we recognize three arguments responsible for the production of a performance: " The analysis A (which may be a complex combination of different special analyses), " the combinatorial stemma, i.e. the stemma's tree structure, and " the performance grammar, i.e. the functional dependence of the operators from the analysis A as well as their action type on the stemma's mothers and daughters. It is in this restricted, but precise context that we are going to discuss the problem of inverse performance. 3 The Problem of Inverse Performance To fix the ideas, we select a score throughout this discussion. Suppose that it is initialized by the primavista mother LPS. With this in mind, we have a performance mapping diagram: {I Performance Grammars } X { Analyses } X { Combinatorial Stemmata } -*P{EerormncsM BUILDPERFORMANCESCORE 4 FRANEÂ~R { Performance Scores } NERTPROMNECR 534 I C M C P RO C EE D I N G-S 1995

Page 535 ï~~The mapping factorizes through the building of a performance score which is integrated to yield the resulting Performance = j PerformanceScore. To describe, which triples lie above the given Performance, we take the inverse image of Performance under PERFORM, and we may call this set the critical fiber, or the fiber of the critiques of Performance. CRITIQUES (Performance) = PERFORM -1 (Performance). An inverse performance of Performance is just a selective function INVERSE such that (Grammar, Analysis, Stemma) = INVERSE (Performance) E CRITIQUES (Performance). The general problem of describing the critiques of a performance P depends upon the local choice of admissible objects within the fiber space. To boil down the manifold of solutions to reasonable complexity, one is constrained to cover the total space by "local" charts. This means that we have to select well-defined local submanifolds of grammars, analyses and stemmata: selected CRITIQUES chart inclusion I PERFORM S local global fiber localfiber local PERFORM all performances -". Performance " all performances 4 The Three Dimensions of a Critique We now want to make local manifolds of critiques more explicit with regard to its three constituents: grammar, analysis and stemma. Let us start with the most elementary one: 4.1 Combinatorial Complexity Reconstruction of critiques from a given performance first of all depends upon the process of its generation from a supposed primavista lecture. This is not a constraint from RUBATO's performance-score structure but an expression of how artists work. You start with a primavista version, then you successively improve your performance by a complex creative feedback process leading to multiple decisions how to define and to refine the parts of a piece where corrections should intervene. The objective shape of this creative process is described by the combinatorial stemma of a performance score. It is a simplification of the "inductive" system of all the LPS, as they are generated by the splitting process from operator actions, in fact a tree of parts of the given score, generated by successive refinements I(i): refinement LPS LPS ps 'Y "-"sub-compositions"of LPS ( LPS, L.PS2 LPSn ) new for 1(1) LPSJ(1 _ _ __sst _ I ( LPS11 LPS12 LPS2 ) new for 1(2) rPSg2; (LPSill LPS211 LPS212 ) new for (n)LPl) IC MC PROCEEDINGS 199553 535

Page 536 ï~~This information is as essential as delicate for the reconstruction of a critique. It is essential because the possible strategies depend heavily on the spectrum of combinatorial stemmata we may consider. The most important factor of this combinatorial complexity is the depth of the stemma, i.e. the total number n = depth(stemma) of refinements. In the case of " locally linear" grammars (see section 5.1), algebraic varieties of critiques of a given performance emerge with a degree proportional to the depth. And it is delicate because a priori, the performance as such does not tell anything about possible underlying stemmata. We only get hints from the fragmentation and grouping of the score, but these are not reliable since a performance may stem from independent, autonomous refinement processes. This aspect is a central concern for understanding what an artist really is doing while developing his performance. -- By ram(LPS), we denote the ramification number at LPSi, i.e. the number of daughters of LPSi. In the sequel, we shall often identify a member LPS1 of a stemma with its kernel (= event set). 4.2 Local and Global Compositions and the Reconstruction of Performance Fields The mathematical structure of such a stemmatic covering of the kernel of the primavista LPS is that of a global composition, as introduced in (Mazzola, G. 1990); it is the very concept of a musical manifold. Recall that in our context, a local composition is a finite subset K of points of a supporting real vector space M. A global composition is a covering of a finite set by local compositions (charts), glued together by isomorphisms of local compositions on the overlaps of these charts. In the classical theory, a morphism of local compositions f: (K, M) -3 (L, N) is a set map f from K to L that extends to an affine map F: M -+ N. To deal with performance transformations, this setting is extended to differentiable contexts. A local spline morphism ffrom K to L is defined to be a "normal" morphism of local compositions f: (K,M) -* (DL, Horn (XM,N)), where M is the symmetric tensor algebra of M, Horn denotes the linear transformations, and DL is a local composition which canonically projects onto L. We say thatf is differentiable of degree d, if it may be extended to a differentiable map F whose derivatives DiF of degrees i = 0,..., d coincide with the ith coordinates of the values off at the points k of K: Proji(f(k)) = DiF(k):MÂ~ -4N. Differentiable global spline compositions are now canonically glued by differentiable local spline isomorphisms, and we understand differentiable global spline morphisms of global spline compositions as being maps which-as usual-restrict to differentiable local spline morphisms on their charts. Example 1. For tempo reconstructions, one considers a series K = { 01, 02,..., on} of increasing symbolic onsets of a score which, under a given performance P, is mapped onto the sequence of "physical" onsets P(K) = {P1' P2'" p.p}, see illustration below. The series K is usually covered by the n-1 two-element charts Ki = { oi, o,+1 }, and one obtains two global compositions: K5 and P(K)1. The problem then amounts to finding a differentiable spline morphism sitting over these global compositions such that it boils down to the set-theoretic map from K onto P(K). degree zero solution degree one solution p(K)t........................."p(K);........................................................ locally constan empo locally continuous t~mp \_- i, i /.,",., -.............. 3-' "v............ - -- K.....K.. 536 I C M C P R OC E E D I N G S 199 5

Page 537 ï~~The trivial case of degree zero morphisms can be solved by locally affine (i.e. piecewise linear) morphisms on the charts. This brute choice corresponds to locally constant tempo as it has been considered by (Gottschewski, H.), for instance. Here is a general result including this one-dimensional technique: Theorem 1. (i) If the local composition K = {Xo,...,Xm) is an m-dimensional simplex, then there is an affine Pt extending any given discrete performance map P: K -4 P(K); Pt is uniquely determined on the affine space generated by K. - If the local composition K = {Xo,...,Xm } consists of any m+ I different points in n-space, then there is an extension Pt:Rn - Rnof any given discrete performance map P such that all its coordinate projections are polynomials of degree <m. For global morphisms of degree one, the first derivatives at the onsets oi, di:R - R, are usually prescribed as being the reciprocal values of the corresponding tempi. Then, finding a solution means finding a spline curve with prescribed slopes di (di(1), to be precise) at the onsets of K. Each choice of a function class of local solutions for this differentiable spline morphism defines a particular variety of extensions of the local spline morphisms. It is not recommended to play down this multiplicity of choice since the discrete values of an "experimentally measured" performance may hide essentials about the artist's intentions. The works of (Repp, B.), (Sundberg, J. & Verillo, V.) and (Mazzola, G. & Zahorka, O. 1994a) make more sophisticated proposals and alternatives on tempo curves dealing with interpolations of differentiable spline morphisms for tempo reconstruction. The reconstruction of a differentiable global spline morphism is only the first step backwards to the reconstruction of a stemmatic system of local performance fields-the essence of the performance score through which the performance transformation factorizes. The type of a performance field Y yielding a differentiable extension Pt of P, i.e. Pt = J Y, relates to the class of differentiable functions we are admitting. This is mediated by the formula -1 Y = (JPt) A, the inverse Jacobian of the constant diagonal field A = (1,1,...,1) under Pt, see (Mazzola, G. & Zahorka, 0. 1994a). Example 2. Generalizing the locally linear tempo curves of the prestoÂ~ composition software (Mazzola, G.), one may consider affine fields Y(X) = A*X+B, A being a regular matrix. Here, integration leads to linear ordinary differential equations with well-known integral curves of the generic form j (t) = exp(A-t)....Thus, for the mentioned one-dimensional case of tempo curves, the physical time is a logarithmic function of the symbolic time. This setting is the degree one refinement of the brute force method of locally constant tempi. On RUBATO, more general fields have been introduced which stem from Lie derivatives and cubic splines. But the reconstruction of individual performance fields is not the only problem in the choice of classes for local performance fields, because a stemma is not restricted to its final ramifications (responsible for the actual performance). It is equally a result of a multi-layered propagation of fields from a primavista field by successive interventions of expressive operators. This means that, if it is decided to stick to a defined type of local fields resp. local performance transformations, this decision should also be stable under the expressive operators: Field types should be inherited under sexual propagation. 4.3 Relating Analyses to Performance Scores Whereas analyses are rather freely used to construct a performance score with "direct" performance, inverse performance should be given a more systematic account of the underlying strategies: If everything is allowed, nothing can be said. Hence, we want to assume that for every member Ki of a given I C M C P R O C E E D I N G S 1995 537

Page 538 ï~~stemma, we may apply a kind of restriction AIK of the given analysis A of the entire composition K. In more mathematical terms, we suppose that we are provided with a contravariant set-valued functor, i.e. an assignment defining a set An(K) of analyses of any local composition K, together with a natural mapping An(f ): An(K2). An(K1) for any morphism ft K l- K2 of local compositions, i.e. A anti-commutes with composition of morphisms: An(f)-An( g ) = An( g-f). For a given analysis A in An(K2), we write AIK1 instead of An(f)(A) iff is the inclusion. Functoriality is a requirement of consistency: It expresses the fact that we are using a concept of analysis that is not isolated from the category of local compositions where it is applied to. The artist should be given an analytical frame where it is natural to make use of the given analysis of the entire piece when building the charts of the local performance score. This seems to be a severe restriction of the freedom of choice. But as soon as we deal with the inverse problem, the critical fiber has to start with assumptions of reasonable behavior, the contrary being incompatible with what-after all--a reasonable critique should aim at. Example 3. RUBATO's Metro-, Melo- and HarmoRUBETTES are examples of such functorial analyses. These tools are constructed to produce weights, i.e. real-valued functions on the discrete set of Kevents. RUBATO's WeightWatcher, which manages the splining of these discrete weights, defines differentiable functions on the rectangular frames containing K. This means that there is a natural restriction of such splines to smaller frames associated with subsets of K, given by the LPS of the stemma. This allows to take the restriction and its iterations when starting from such an analysis of the total composition. Attention: One should not confuse the restriction of an analysis with the analysis of the subcomposition. A metrical analysis of a sub-composition may be much coarser than the restriction of the overall metrical analysis. 5 Performance Grammars Suppose that we are given the analysis A of K which gives rise to a "projective" analysis system (AIK)i on our stemma. In the above Example 3, this may be the analysis A = (Ametro, Aeo10, Aharmo) consisting of respective splines and their restrictions. The question of performance grammars is related to the methods resp. operators which the artist uses to generate the succession of all the local performance fields of the stemma when starting from the primavista field on K. We may then make the hypothesis that we are given a general method Q which is a function of a given analysis B of a local composition L, and whose values f (B) are operators that "deform" local performance fields Y on that local composition to yield a new fieldQS2(B) Â~"Y: frame/ frame At this point, we have to pay attention to the above condition that the operator has to conserve the type of field we have decided to deal with, when reconstructing a "field patch" from the given performance. Example 4. Suppose that we are given a weight spline B, for example the melodic or harmonic weight spline. Identify D(B) with the scalar multiplication with the mean value D2(B) of B over the local composition's frame. Then, any type of performance field will be deformed by simple dilatation of its arrows. A constant field will remain what it is, a polynomial one too, an affine one also, etc. However, the restriction of a deformed field will be different from the deformation of the restricted field by the action of the restricted weight! 538 I C MC P R O C EE D I N G S 1995

Page 539 ï~~Next, we want to generate combinations of deformed fields when generating the daughters L1,..., La of a local chart L. Here is the musical motivation: Suppose that you have shaped the tempo of a sequence L of n bars L1,..., Ln within a composition by use of an operator which evaluates a harmonic analysis of the piece. The next step would be to refine this tempo to local variations on each bar Li. In this situation, you may want to take into account the restricted analysis on the ith bar, but you also want to respect the development of the harmonic analysis from the preceding to the subsequent bars. For instance, you want to give your performance an intentional character, e.g. by steering towards a tonal modulation after the ith bar. Then, you are obliged to consider combinations of all these daughter deformations: 02(A) Li) "371 Li i = 1...n. To achieve this goal, denote by fr(i) the frame of Ll. Then we have the evident affine transfer fi of arguments from the frame fr(j) to frame fr(i). We get a transferred field for each point x in fr(j) and 3 on fr(i): 1 ji (x) = (f ji(x)). This technique may be used to construct linear combinations of daughter fields on a fixed daughter frame fry): 3j(x) = cji. (Dl(AILi)!I L,)Iji(x) xe fr(J). This allows to consider linear maps A = (cji) in order to produce weighted combinations of a distribution of local actions. The fact of taking an intentional point of view as sketched above is now reflected in the choice of an upper triangular matrix A, whereas a causal point of view will be seen from a lower triangular matrix. - Let us finally give a precise definition of a locally linear grammar: Definition 1. Given a stemma (Ki) of a composition K, a family A. = (A(i,...)) of ram(Ki)-by-ram(Ki) matrices-possibly depending on further parameters of the stemma-is a locally linear grammar for this stemma. The locally linear grammar is constant ifWno other parameters except i occur We say that the matrix coefficients of A. are the grammar's coefficients. To consider a locally linear grammar means to admit a determined combination of localization methods defined by S2 This defines a range of a priori strategies in order to produce a final performance from given analytical deformation operators. 5.1 Algebraic Varieties of Locally Linear Grammars We now want to look at the critical fibers defined by this method for concrete examples. For the following fundamental shaping operators, the operator 0 (A) " 3 is linear in 3: " the above scalar multiplication, Â~ (A) is a real number; " the scalar operator in RUBATO, where Q(A) = A, i.e. Q2 (A) " (x) = A (x) - 3 (x); " the Lie basis operator of RUBATO, with the Lie derivative L and an affine transformation Af. 9 (A) " 3 (x) = 3 (x) + LilogA (x) -Aff(x). In this case, the following is immediate: Theorem 2. For a locally linear grammar, the perfo rmance fields on the final (output) charts are polynomials in the grammar's coefficients. Their coefficients are determined by the operator, the stemma and the field of the primavista LPS by successive cycles of restriction, followed by the local operator action, and transfer to sister frames. ICMC PROCEEDINGS 199553 539

Page 540 ï~~An immediate consequence of this general fact in case of constant local fields and constant locally linear grammars is this: Theorem 3. The critical fiber of a performance with constant LPS fields, linear operators and locally constant grammars on a fixed stemma is an affine algebraic variety, the grammatical variety of the performance, defined by polynomials of degree = depth(stemma). In this situation, one may look at different performances with locally constant fields, and corresponding grammatical varieties. Their intersection then determines the set of common grammars, i.e. of grammatical components of INVERSE values. These grammars are found by common solutions of a (finite) number of polynomials, i.e. real-valued points of an affine variety. 6 Consequences for Musical Aesthetics and Criticism The possibility to exhibit critical fibers resp. algebraic grammatical varieties for given performances on the very "easy" level of locally constant performance fields has profound consequences for the classification problem of performances: One is now in state to ask whether the grammatical varieties of two performances are isomorphic or not, or whether one is a deformation/specialization of the other. The classification of varieties now becomes a work of musicological relevance. And comparative criticism will now become a field of precise research and no longer a sector of literature. 7 Summary By exhibiting the structure of performance transformations under RUBATO's methodology, we are able to describe the critical fiber and the grammars of performances as being algebraic varieties in the case of locally constant performance fields, i.e. of locally affine performance transformations. This opens a rich field of research on comparative criticism by use of classification theory of algebraic varieties. 8 References Gabrielsson, A.: Expressive Intention and Performance. In: Music and the Mind Machine. Steinberg, R. ed., Springer, Berlin 1995. Gottschewski, H.: Tempoarchitektur. Musiktheorie, Heft 2, 1993. Mazzola, G.: Geometric der Tone. Birkhauser, Basel 1990. Mazzola, G. & Zahorka. O. 1994a: Tempo Curves Revisited: Hierarchies of Performance Fields. Computer Music Journal. Vol. 18, 1, 1994. Mazzola, G. & Zahorka. O. 1994b: The RUBATO Performance Workstation on NeXTSTEP. In: Proceedings of the ICMC 1994, Arhus 1994. Repp, B.: Diversity and Communality in Music Performance: An Analysis of Timing Microstructure in Schumann's Traumerei. J. Acoust. Soc. Am. Vol. 92, 1992. Sundberg, J. (ed.): Generative Grammars for Music Performance. KTH, Stockholm 1994. Sundberg, J. & Verillo, V.: On the Anatomy of the Retard: A Study of Timing in Music. I. Acoust. Soc. Am. Vol. 68, 1980. Zahorka, 0.: PrediBase--Controlling Semantics of Symbolic Structures in Music. ICMC Proceedings, Banif 1995. 540 ICMC PROCEEDINGS 1995