Page  369 ï~~Knowledge Representation of Sounds and Sonic-Structures Based on Constraints and Multiple Inheritance. Peter Lunden Royal Institute of Technology (KTH) Dept. of Speech Communication and Music Acoustics Box 700 14 S-100 44 Stockholm Sweden email: A new system for knowledge representation of sound and sonic structures is described. The system is based on a taxonomy of sounds and sonic structures and is implemented in a classless multiple inheritance object-oriented system based on the prototype paradigm. Multiple inheritance facilitates the partitioning of different notions of sounds into there own distinct inheritance paths and greatly increases the reusability of knowledge in the system. Constraints are applied to describe dependencies and dynamic behaviour of sounds and sonic structures. The system can be used to build hierarchical models of sonic-structures with complex dynamic behaviour and complicated interdependencies which makes it useful in a computer music environment. 1. introduction Not very many researchers has tried to opposed the problem of how to represent knowledge about sounds and sonic-structures in a generalised way. Most of the works in this area are closely related to a specific synthesis or analysis/resynthesis method and are almost useless for in other situations. This is the case in most computer music systems today which means that the knowledge gained by a composer working with such system is non-accumulative, the knowledge may become useless when the system is changed or when he moves to another system, which he probably would like to do because no single system exists that can do everything a composer would like to do, not even during the work with one single composition. This is one of the greatest problems for composers working with computer music systems today. It would be a great advantage if the knowledge was accumulative and could be use across many different system. A number of things has to be accomplished to achieve this. Computer music systems should employ acoustic and psychoacoustic terminology as mush as possible and not obscure "Yamaha units". The knowledge must be expressed in a generalised form and not tied to a particular synthesis method or system. This may be trivial when it comes to parameters like frequency and amplitude but it becomes much more difficult with timbre and temporal behaviour. The system described here tries to tackle this problems. 2. Design and Implementation The current system is implemented in the CCPL environment [Lunden 89] using an extended version of the KR knowledge representation system [Giuse 89] which is a classless object oriented system based on the prototype, frame and constraint paradigms. KR is a part of the Garnet project [Myers et al 901. In KR every object may serve as a prototype of new objects (as a class in class-based object-systems). The word object, prototype and frame have the same meaning and they may be interchanged but prototype is preferred. The object slots or instance-variables are called sound parameters or just parameters. As there are no distinction between classes and objects in a prototype based system, new sound prototypes may be defined in terms of other prototypes, the user may define new prototypes by refinement of existing prototypes redefined in the system or made earlier by the user. The frames paradigm facilitates multiple inheritance and distributed knowledge (within the system). This make it possible to divide the sound space into domains where different notions of sound can have there own distinct inheritance paths which greatly increases the reusability. Constraints are used to describe dependencies between sound parameters and can be used to define the temporal behaviour of prototypes. Constraints add much power and expressiveness to the system. ICMC Proceedings 1993 369 2P.11

Page  370 ï~~3. Typology of Sounds A typology of sounds is the foundation of this knowledge representation system. Several typologies has been proposed by [Schaeffer 66], [Smalley 86] and others but they are not very formal and are therefore no well suited to be incorporated into a computer music system. The typology used in this system is designed by the author based on ideas from Schaeffer and Smalley. The inheritance structure of the knowledge system is closely reflecting this typology. The sound space is divided in domains, where each domain should be as independent of the other domains as possible (ortogonality). Adomain may be future divided into subdomains. Each domain is represented by a prototype where the domain-subdomain relationship defines the inheritance path. 4. Inheritance Structure The most simple domains are the Pitch-Domain and the Loudness-Domain which consists of only one dimension except the time-dimension which is considered as the fundamental dimension of all domains. There are not further subdivision of this domains. 4.1. The Spectral-Domain The Spectral-Domain is multidimensional and give raise to many questions of how it should be represented. It is future divided into several subdomains (See figure 1). 4.1.1. Spectral-Envelop The Spectral-Envelop domain describes the dependency between amplitude and frequency of the sound. It has two subdomains, the LowpassSpectral-Envelop which describes the spectral envelop by a pseudo lowpass filter and the Formants-Spectral-Envelop where the envelop is defined by a set offormants. In the future there will be subdomains like LPC-Spectral-Envelop where the spectral envelop is determined by time-varying LPC coefficients. 4.1.2. Discrete-Spectrum Sounds with a partial structure should inherit from the subdomains of Discrete-Spectrum. This domains manages the frequency of the partials but leaves the amplitude to be handled by the Spectral-Envelop domain. It may seams as a superficial division but it greatly influences the reusability. The Partial-Distribution describes the partial ratios. It has several subdomains which are: Harmonic-Partials where the ratios are taken from the harmonic series, Enumerated-Partial were the ratios are listed and Computed-Partials where they are computed. Partial-Distribution domain is supposed to work together with the Partial-Selection subdomains which are selecting partials from the set defined by the PartialDistribution domain. Every-Partial and RandomPartials are subdomains of Partial-Selection and are well described by there names and need no further description. The Partial-Duration describes the duration of partial. It is used to model sounds where the partials has different decay times, i.e. sounds from membranes or bars. 4.1.3. Continuos-Spectrum Sounds without a partial structure should inherit from the Continuous-Spectrum domain. Noise is currently it's only subdomain. 4.2. Temporal-Domain The temporal-domain describes the temporal behaviour of the sound. It's subdomains are Iteration, Continuous and Impulse. 5. Sounding Prototypes The prototypes described so far can not produce any sounds. To be able to produce any sounding result the prototype has to inherit knowledge from a synthesiser interface prototype. The purpose of such prototype is to act as an interface between the knowledge system and the synthesiser software. There are currently only one synthesiser (interface and programme) implemented in the system, a FOF-synthesiser spectral-domain discrete-spectrum spectral-envelop partial-selection lowpass-spectral-envelop every-partial random-partials partial-distribution formant-spectral-envelop harmonic-partials computed-partials enumerated-partials quantized-partials stretched-partials Figure 1. Structure of the Spectral-Domain. 2P.11 370 ICMC Proceedings 1993

Page  371 ï~~[Rodet 85]. It will be easy to incorporate other synthesiser algorithms like additive synthesis or subtractive synthesis (with real filters) but rather difficult to implement methods like FM synthesis where the parameters are not so closely related to the acoustic result. A specific sound prototype is defined by a set of parameters, a set of methods and an inheritance path. As an example, suppose the composer would like to define a prototype for vowels. This prototype will need knowledge about amplitude and frequency which is gained by including the PitchDomain and Loudness-Domain prototypes in the inheritance path. The spectral envelop of this sound can be obtained from the FormantsSpectral-Envelop which knows things like formant frequency and formant bandwidth. If the prototype in mind is a model a real voice then it should inherit from Harmonic-Partials to define it's partial-structure.. 6. Dependencies Constraints are a very elegant solution to the problem of describing dynamic dependencies between parameters. To describe a dependency it is just to include the parameter ofwhich it should depend upon in the definition of a constraint (a constraint is an expression which is recomputed every time any of it's components are changed) and assign the constraint to dependent parameter. 7. Temporal Behaviour As every domain includes the notion of time it is very easy to define the time-dependency of a parameter, just include a time-parameter in the constraint assigned to the parameter. There are several time-parameters: local-time which hold the local time, the relative-time which goes from 0 to 1 during the lifetime of the sound and globaltime which are the current time of the whole piece. There are other mechanisms than constraints in the system that can be used to define the temporal behaviour of sound prototypes. The most important is the Each-Time cycle mechanism inherited from the CCPL environment [Lund~n 911. It is a set of methods which are called at certain points in time during the life of a sound. An Each-Time cycle must be started every time a sound prototype needs attention: An Each-Time cycle may be scheduled by a call to the Delay method. Each cycle except the first one consists of calls to the pre-each-time, the each-time and the post-each-time methods. The very first cycle additionally includes a call to the First-Time method which is called just before the sound is started and before the other methods are called. It is used for initialisation purposes. The typical use of this mechanism is demonstrated by the Iteration prototype. Every time an iteration should occur one Each-Time cycle is trigged. A copy of the prototype to iterate is schedule with zero delay and a new Each-Time cycle is schedule for the next time an iteration should happen. 8. Summary Knowledge of sounds and sonic-structures can be represented in a general form in a network of frames. The knowledge may be shared by many prototypes and thereby gaining reusability the system. This is achieved by using a frame-based system with a carefully designed inheritance structure. The prototype paradigm facilitates design of new sounds by refinement. The utilisation of constraints make it easy do describe complex interdependencies between different sound-parameters and different sonicstructures. Constraints are also used to define time-dependencies of parameters. References [Giuse 89]. Giuse, D. A.: "Efficient Knowledge Representation.", Technical Report CMU-CS-89 -142, Carnegie Mellon University Computer Science Depatment, 1989 [Lunden 89]. Lunden, P.: "CPL: a Composers View of Computer Programming". KACOR report 13/89. Royal Institute of Technology, Dept. of Speech Communication and Music Acoustics. [Lunden 91]. Lunden P.: "Sound-Models: the Representation of Knowledge about Sound-Synthesis in the CPL Environment." ICMC 1991 Proceedings, International Computer Music Association, San Francisco, 1991. [Myers et al90]. Myers Brad A., Giuse Dario A., Dannenberg Roger B.,Vander Zanden Brad, Kosbie David S., Pervin Edward, Mickish Andrew and Marchal Philippe: "Garnet: Comprehensive Support for Graphical, HighlyInteractive User Interfaces.". IEEE Computer 23(11):71-85, 1990. [Rodet 85]. Rodet, X. "Tie-Domain FormantWave-Function Synthesis". Computer Music Journal nr. 3 1985. [Schaeffer 66]. Schaeffer P.: "Trait6 des Objets Musicaux.". Seuil, Paris 1966. [Smalley 86]. Smalley Denis: "SpectroMorphology and Structuring Processes." in the Language of Electroacoustic Music. Emmerson, S. ed. Macmillan Press, 1986. ICMC Proceedings 1993 371 2P.11