Page  00000052 PIECE: A Music Language Editor Synchronized with Graphical Views Satoshi Nishimura School of Computer Science and Engineering The University of Aizu Aizu-Wakamatsu city, Fukushima, 965-8580, Japan Email: nisim@u-aizu. ac. jp Abstract This paper describes a music editing system called the PMML Integrated Emacs-based Composing Environment (PIECE). The system provides both a textual-language view and piano-roll view synchronized with each other. The consistency between the views is maintained even if control structures or macros are used in the language texts. A novel algorithm for maintaining the consistency is presented. 1 Introduction In MIDI sequencers, there are two forms of music representation through which music is edited. One is graphical representation such as piano rolls or conventional music notations. The advantage of graphical representation lies in intuitiveness; one can easily understand the correspondence between actual music and the representation. On the other hand, graphical representation is weak in structured or algorithmic description. The other form is textual representation such as music languages. Although textual representation is inferior in intuitiveness, it is the most suitable for structured/algorithmic description. To make use of all the advantages, this paper treats with a music editing system supporting both. of the forms. The system contains both a text editor specialized for a music language and a graphical editor based on a piano roll which is unstructured, time-linear representation of music (Figure 1). As the music language, the PMML [1] is employed; however, some of techniques described in this paper are also applicable to other languages. The text editor is the GNU Emacs editor extended for the PMML, through which users can edit the text in a usual way. On the piano-roll editor, users can modify pitch, onset time, duration, velocity, and control values by dragging "note bars" or "value dots" with pointing devices. In the above system, the synchronization between the textual and graphical views is maintained. For example, clicking a note bar in the piano roll moves the cursor to the corresponding position in the text window. Conversely, issuing a synchronization command at some location in the text window scrolls the piano roll to the corresponding position. In addition, modification in the language text is reflected in the piano roll, and vice versa. For example, when the text is changed and then the synchronization command is issued, the piano roll is updated correctly. Conversely, as illustrated in Figure 1, dragging a note bar or value dot in the piano roll adjusts the text appropriately. There are many commercial MIDI sequencers supporting the synchronization of graphical views and textual views [2]; however, textual views in those systems are simple time-linear lists of events. The system in this paper differs in that control structures or macros can be used in the texts, and as a result, it enables us to modify note attributes or control values from the graphical view with keeping the algorithmic structures of music. Language View C D E Change 'E' to 'F' Change 'D' to 'D(v-=20)' Graphical View - "l Drag E - D pitch SloDragty f J I __ I f velocity Stime Figure 1: Music editor with synchronized views 2 Features of the PMML The PMML is a music description/manipulation language designed for MIDI sequencing. It supports both direct music description and algorithmic representation. The PMML has a feature for specifying the attribute values of a particular note relatively to their default values in addition to specifying them absolutely. This feature is very useful in reflecting changes in the graphical view to the text. For example, if the velocity of the note described by - 52 - ICMC Proceedings 1999

Page  00000053 repeat (2) ( C4 D4 Drag Change 'C4' to 'C Move in parallel (a) Example 1: Loops 4 0 C4 5 C4 (D4) D4) n (a) Example 1 C5 D5 dt nv du dtn v du C4(b) Exampl04 (b) Example 2 dt: onset time n: note number v: velocity du: duration (b) Example 2: Subordinated notes v (scale=2) c ~ v (scale=2) c~ (c) Example 3: Attribute-transforming effectors Figure 2: Examples of synchronization 'D' is changed from 80 to 60 through the graphical view, there are two possible modified results of the command: the absolute form 'D(v=60)' and relative form 'D(v-=20) b The relative form is superior because mutual velocity relationships among notes are still maintained when the default value is altered later, or when the same phrase is reused with a different default value. In our system, the relative form is used whenever the original text is not written in the absolute form. In the PMML, music can be manipulated through event-processing modules called effectors. Effectors can also be applied for generating notes bound to other notes, in other words, notes that inherit some of their attributes from other notes. Those notes are called subordinated notes. A typical example of a subordinated note is a grace note. This feature is useful in describing music; however, it causes some difficulties in synchronization between the text and graphical views. 3 Event Updating Algorithm The most difficult case of synchronization arises when a note bar or value dot in the graphical view is dragged. This section describes an algorithm for updating the text and graphical views appropriately in such a case. Since the PMML allows the use of macros and control structures, updating only the dragged graphical symbol and its corresponding text may cause inconsistency as explained below. In the following description, an event means either a note.or a MIDI non-note (e.g. control-change) message. (c) Example 3 Figure 3: Event dependency graphs corresponding to the examples in Figure 2 There are two sources of such inconsistency. One is one-to-many relationships between an eventgenerating command in the text and symbols in the piano roll. An example for this is shown in Figure 2(a). Due to the loop structure, each note command has two corresponding bars in the piano roll, and therefore, dragging one bar requires not only updating the command text but also moving the other bar to maintain consistency. The other source of the inconsistency is subordinated notes. In Figure 2(b), because the effector 'addm.otes(n+=12)' is attached, each note accompanies a subordinated note whose pitch is an octave higher. Also in this case, dragging one note bar should move the other in parallel. Effectors which transform note attributes will cause another type of difficulty, as exemplified in Figure 2(c). The effector 'modify.notes(v*=2)' will scale the velocity of each note by two. Therefore, when the velocity of the note is increased by 20 from the graphical view, the correct way to adjust the text is increasing the velocity by 10. The above last example requires the inversion of the transformation process, and of course, it is impossible to handle a general case. In our system, we treat with only the transformations described by a linear expression ax + b (a # 0). For other transformations, modification from the graphical view with respect to that attribute is inhibited. To identify which graphical symbol should be updated and how the text is adjusted, we introduce a directed graph G = (V, E) called an event dependency graph. Each vertex of G is an event vertex, S-vertex- (synchronization vertex), or U-vertex ICMC Proceedings 1999 - 53 -

Page  00000054 ("unknown" vertex). Each edge of G is either unlabeled or labeled by a pair of an attribute name (onset time, note number, velocity, duration, or control value) and a scale value. The event dependency graph is constructed as follows. For each event in the graphical view, an event vertex is created. When an event-generating command is executed more than once and thus generates multiple events, an S-vertex is created, and then, unlabeled edges from/to each of the events to/from the S-vertex are placed (Figure 3(a)). If an attribute of an event depends on that of another event, an edge labeled by the attribute name is created (Figure 3(b)). The scale value of the edge is set to a when the dependency is represented as ax + b (In Figure 3, the scale value is not shown if it is unity). In the case of attribute-transforming effectors, the labeled edge will be self-looped (Figure 3(c)). When the dependency is not linear, or when the PMML compiler fails to identify the destination of the dependency (this occurs at least when an attribute value is assigned to a non-local variable in an effector definition), a U-vertex is created, to which a labeled edge is directed. The algorithm for updating the text and graphical views when an attribute X of an event v is increased by d is as follows: Step 1. Let W be the set of vertices having such a path p to the vertex v that p consists of only Xlabeled edges (v E W). Then, remove those vertices from W whose event-generating command resides in a system macro library. Step 2. Choose a vertex w from W, based on the user's preference. Step 3. Check if there is a vertex having a path from w and also having an X-labeled edge to a Uvertex. If this is true, consider this modification as impossible and exit. Step 4. For every event vertex x having a path from w, adjust the symbol corresponding to x in the graphical view so that X increases by d S(w, x)/S(w, v), where S(s, t) represents the product of the scale values of all the X-labeled edges along the path from s to t (for a self-looping edge, its scale value must be multiplied just once). Step 5. Adjust the event-generating command in the text corresponding to w so that X increases by d/S(w, v). In the above algorithm, Step 4 can be repeated during the dragging, by which rubber-banding is realized. This makes the two note bars in Figure 2(a) or Figure 2(b) felt as if they are tied with each other. Figure 4 shows another example containing grace notes and Figure 5 depicts its corresponding event dependency graph. Suppose that the velocity of the D grace note represented by Vertex (2) is increased by 10. Step 1 of the above algorithm will for($i,l,2) { // repeat 2 times // set the default velocity // (first time: 40, second time: 50) v = ($i == 1)? 40: 50 // play a note with a grace note // with letting the velocity of the // grace note to be the half of the // original one c(grace(D(v*=0.5))) ) // attach an effector for clipping velocity modify.notes(v= (v>=80)? 80: v) E // play an E note Figure 4: An example containing grace notes velocity before editing i velocity after editing 20130 S 25135 D D (2) (4) v n dtna nd U (scale-O.5) (scale-0.5) C c (Es (1V (3) 40160 50/70 Figure 5: Event dependency graph corresponding to Figure 4 determine W as Vertices (1) and (2). If Vertex (1) is chosen in Step 2, the velocity values of Vertices (1) through (4) will be changed as indicated in Figure 5. In this case, the resultant text will be 'C(grace(D(v*=0.5))) (v+=20)'. If Vertex (2) is taken in Step 2, only the velocity of Vertices (2) and (4) will be changed, and the resultant text will be 'C(grace(D(v*=0.5) (v+=10)))'. In either case, the synchronization between the two views are correctly maintained. Meanwhile, the modification of the E note from the graphical view will be inhibited. 4 Implementation The PIECE system is implemented using three processes of the Unix operating system (Figure 6). Currently, the system runs under the Linux or SGI/IRIX operation system. The Emacs process is the GNU Emacs editor extended with Emacs Lisp codes. The MIDI server process, developed with the C language, is responsible for MIDI input/output and time management. The graphics editor process, developed with the Java language, displays the piano-roll view and handles modification requests from the user. These three processes are connected by pipes, which transmit controlling information such as playing start/stop requests, po -54 - ICMC Proceedings 1999

Page  00000055 MIDI I/O Normal chunks of the Standard MIDI file string table chunk file table chunk macro node chunk event node chunks S macro caling tree!. sourc Fie Io une Number Column Number Maacro Namen i T Parent Node D10 ' / Source Fie 10 Li Une Number Column Number SParent Node ID Sn Data Dependency Flaoa Per-evenvt inormation scale vaes(otna text-based editing Spiano-roll based user editing Figure 6: System structure of the PIECE sitioning requests, text-highlighting requests, and text-adjustment requests. MIDI event data as well as information about source positions and event dependencies are provided to the graphical editor and MIDI server processes via an extended MIDI file. The extended MIDI file is created by the PMML compiler upon a request from the Emacs process. The structure of the extended MIDI file is illustrated in Figure 7. To include additional information in the Standard MIDI File with keeping compatibility, we appended non-standard data chunks at the bottom of the Standard MIDI File.' The string table chunk contains the strings of macro and file names. The file table chunk specifies the mapping between source file IDs and file names. The macro node chunk contains a tree-formed data structure representing the macro calling sequence to each event-generating command. This information enables the identification of every related position in the text when a note bar is clicked in the piano roll. The event node chunks encode the information of the event dependency graph, in which the S-vertices are implicitly expressed by that two or more events have the same source position. The total size of the extended MIDI file is typically twice as large as the Standard MIDI file. Figure 8 shows a screen snapshot of the running system. When the user issues a synchronization command in Emacs, the Emacs process updates the extended MIDI file using the PMML compiler, and then sends a positioning request to the graphical editor process so that the piano roll scrolls to the position corresponding to the text cursor location. When an event is dragged in the piano-roll view, the graphics editor process executes the event up1 The approach maintains the compatibility because by specification non-standard chunks are to be ignored. As compared with the meta-event embedding approach as in NoTAMIDI [3] or Expressive MIDI (4], this approach reduces the file size and also advantageous in that it does not interfere standard chunks. Figure 7: Structure of the extended MIDI file Figure 8: A screen snapshot of the PIECE dating algorithm described in Section 3, and sends a text-adjustment request to the Emacs process, which then does necessary text updates. 5 Summary The paper described a newly-developed music editing system named PIECE. By providing both a textual-language view and piano-roll view synchronized with each other, the system enables the structured or algorithmic description of music without sacrificing intuitiveness. In the future, we would like to add the synchronized views of conventional music notations into the system. References [1] S. Nishimura, "PMML: A Music Description Language Supporting Algorithmic Representation of Musical Expression," in Proceedings of ICMC98, pp. 171-174, 1998. [2] C. Yavelow, Music & Sound Bible, ch. 17. IDG Books, 1992. [3] K. E. Nordli, "NoTAMIDI Meta-Events," in Beyond MIDI (E. Selfrideg-Field, ed.), pp. 73-79, MIT Press, 1997. (4) D. Cooper, K.-C. Ng, and R. D. Boyle, "Expressive MIDI," in Beyond MIDI (E. Selfrideg-Field, ed.), pp. 80 -98, MIT Press, 1997. ICMC Proceedings 1999 -55 -